Video data stream concept

ABSTRACT

Decoder retrieval timing information, ROI information and tile identification information are conveyed within a video data stream at a level which allows for an easy access by network entities such as MANEs or decoder. In order to reach such a level, information of such types are conveyed within a video data stream by way of packets interspersed into packets of access units of a video data stream. In accordance with an embodiment, the interspersed packets are of a removable packet type, i.e. the removal of these interspersed packets maintains the decoder&#39;s ability to completely recover the video content conveyed via the video data stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/392,785 filed Apr. 24, 2019, which is a continuation of U.S.patent application Ser. No. 15/928,742, filed Mar. 22, 2018, now U.S.Pat. No. 10,484,716, which is a continuation of U.S. patent applicationSer. No. 14/578,814, filed Dec. 22, 2014, now U.S. Pat. No. 9,973,781,which is a continuation of International Application PCT/EP2013/063853,filed Jul. 1, 2013, which claims priority from U.S. Patent Application61/666,185, filed Jun. 29, 2012, all of which are incorporated herein intheir entireties.

The present application is concerned with video data stream conceptswhich are, in particular, advantageous in connection with low delayapplications.

BACKGROUND OF THE INVENTION

HEVC [2] allows for different means of High Level Syntax signaling tothe application layer. Such means are the NAL unit header, ParameterSets and Supplemental Enhancement Information (SEI) Messages. The latterare not used in the decoding process. Other means of High Level Syntaxsignaling originate from respective transport protocol specificationssuch as MPEG2 Transport Protocol [3] or the Realtime Transport Protocol[4], and its payload specific specifications, for example therecommendations for H.264/AVC [5], scalable video coding (SVC) [6] orHEVC [7]. Such transports protocols may introduce High Level signalingthat employs similar structures and mechanism as the High Levelsignaling of the respective application layer codec spec, e.g. HEVC [2].One example of such signaling is the Payload Content ScalabilityInformation (PACSI) NAL unit as described in [6] that providessupplementary information for the transport layer.

For parameter sets, HEVC includes Video Parameter Set (VPS), whichcompiles most important stream information to be used by the applicationlayer at a single and central location. In earlier approaches, thisinformation needed to be gathered from multiple Parameter Sets and NALunit headers.

Prior to the present application, the status of the standard withrespect to Coded Picture Buffer (CPB) operations of HypotheticalReference Decoder (HRD), and all related syntax provided in SequenceParameter Set (SPS)/Video Usability Information (VUI), Picture TimingSEI, Buffering Period SEI as well as the definition of the decodingunit, describing a sub-picture and the syntax of the Dependent Slices aspresent in the slice header as well as the Picture Parameter Set (PPS),were as follows.

In order to allow for low delay CPB operation on sub-picture level,sub-picture CPB operations have been proposed and integrated into theHEVC draft standard 7 JCTVC-I1003 [2]. Here especially, the decodingunit has been defined in section 3 of [2] as:

decoding unit: An access unit or a subset of an access unit. IfSubPicCpbFlag is equal to 0, a decoding unit is an access unit.Otherwise, a decoding unit consists of one or more VCL NAL units in anaccess unit and the associated non-VCL NAL units. For the first VCL NALunit in an access unit, the associated non-VCL NAL units are and thefiller data NAL units, if any, immediately following the first VCL NALunit and all non-VCL NAL units in the access unit that precede the firstVCL NAL unit. For a VCL NAL unit that is not the first VCL NAL unit inan access unit, the associated non-VCL NAL units are the filler data NALunit, if any, immediately following the VCL NAL unit.

In the standard defined up to that time, the “Timing of decoding unitremoval and decoding of decoding unit” has been described and added toAnnex C “Hypothetical reference decoder”. In order to signal sub-picturetiming, the buffering period SEI message and the picture timing SEImessage, as well as the HRD parameters in the VUI have been extended tosupport decoding units, as sub-picture units.

Buffering period SEI message syntax of [2] is shown in FIG. 1.

When NalHrdBpPresentFlag or VclHrdBpPresentFlag are equal to 1, abuffering period SEI message can be associated with any access unit inthe bitstream, and a buffering period SEI message shall be associatedwith each RAP access unit, and with each access unit associated with arecovery point SEI message.

For some applications, the frequent presence of a buffering period SEImessage may be desirable.

A buffering period was specified as the set of access units between twoinstances of the buffering period SEI message in decoding order.

The semantics were as follows:

seq_parameter_set_id specifies the sequence parameter set that containsthe sequence HRD attributes. The value of seq_parameter_set_id shall beequal to the value of seq_parameter_set_id in the picture parameter setreferenced by the primary coded picture associated with the bufferingperiod SEI message. The value of seq_parameter_set_id shall be in therange of 0 to 31, inclusive.

rap_cpb_params_present_flag equal to 1 specifies the presence of theinitial_alt_cpb_removal_delay [SchedSelIdx] andinitial_alt_cpb_removal_delay_offset [SchedSelIdx] syntax elements. Whennot present, the value of rap_cpb_params_present_flag is inferred to beequal to 0. When the associated picture is neither a CRA picture nor aBLA picture, the value of rap_cpb_params_present_flag shall be equal to0.

initial_cpb_removal_delay [SchedSelIdx] andinitial_alt_cpb_removal_delay [SchedSelIdx] specify the initial CPBremoval delays for the SchedSelIdx-th CPB. The syntax elements have alength in bits given by initial_cpb_removal_delay_length_minus1+1, andare in units of a 90 kHz clock. The values of the syntax elements shallnot be equal to 0 and shall not exceed 90000*(CpbSize[SchedSelIdx]÷BitRate [SchedSelIdx]), the time-equivalent of the CPBsize in 90 kHz clock units.

initial_cpb_removal_delay_offset [SchedSelIdx] andinitial_alt_cpb_removal_delay_offset [SchedSelIdx] are used for theSchedSelIdx-th CPB to specify the initial delivery time of coded dataunits to the CPB. The syntax elements have a length in bits given byinitial_cpb_removal_delay_length_minus1+1 and are in units of a 90 kHzclock. These syntax elements are not used by decoders and may be neededonly for the delivery scheduler (HSS).

Over the entire coded video sequence, the sum ofinitial_cpb_removal_delay [SchedSelIdx] andinitial_cpb_removal_delay_offset [SchedSelIdx] shall be constant foreach value of SchedSelIdx, and the sum of initial_alt_cpb_removal_delay[SchedSelIdx] and initial_alt_cpb_removal_delay_offset [SchedSelIdx]shall be constant for each value of SchedSelIdx.

The picture timing SEI message syntax of [2] is shown in FIG. 2.

The syntax of the picture timing SEI message was dependent on thecontent of the sequence parameter set that is active for the codedpicture associated with the picture timing SEI message. However, unlessthe picture timing SEI message of an IDR or BLA access unit is precededby a buffering period SEI message within the same access unit, theactivation of the associated sequence parameter set (and, for IDR or BLApictures that are not the first picture in the bitstream, thedetermination that the coded picture is an IDR picture or a BLA picture)does not occur until the decoding of the first coded slice NAL unit ofthe coded picture. Since the coded slice NAL unit of the coded picturefollows the picture timing SEI message in NAL unit order, there may becases in which it is useful for a decoder to store the RBSP containingthe picture timing SEI message until determining the parameters of thesequence parameter that will be active for the coded picture, and thenperform the parsing of the picture timing SEI message.

The presence of picture timing SEI message in the bitstream wasspecified as follows.

-   -   If CpbDpbDelaysPresentFlag is equal to 1, one picture timing SEI        message shall be present in every access unit of the coded video        sequence.    -   Otherwise (CpbDpbDelaysPresentFlag is equal to 0), no picture        timing SEI messages shall be present in any access unit of the        coded video sequence.

The semantics were defined as follows:

cpb_removal_delay specifies how many clock ticks to wait after removalfrom the CPB of the access unit associated with the most recentbuffering period SEI message in a preceding access unit before removingfrom the buffer the access unit data associated with the picture timingSEI message. This value is also used to calculate an earliest possibletime of arrival of access unit data into the CPB for the HSS. The syntaxelement is a fixed length code whose length in bits is given bycpb_removal_delay_length_minus1+1. The cpb_removal_delay is theremainder of a modulo 2^((cpb_removal_delay_length_minus1+1)) counter.

The value of cpb_removal_delay_length_minus1 that determines the length(in bits) of the syntax element cpb_removal_delay is the value ofcpb_removal_delay_length_minus1 coded in the sequence parameter set thatis active for the primary coded picture associated with the picturetiming SEI message, although cpb_removal_delay specifies a number ofclock ticks relative to the removal time of the preceding access unitcontaining a buffering period SEI message, which may be an access unitof a different coded video sequence.

dpb_output_delay is used to compute the DPB output time of the picture.It specifies how many clock ticks to wait after removal of the lastdecoding unit in an access unit from the CPB before the decoded pictureis output from the DPB.

A picture is not removed from the DPB at its output time when it isstill marked as “used for short-term reference” or “used for long-termreference”.

Only one dpb_output_delay is specified for a decoded picture.

The length of the syntax element dpb_output_delay is given in bits bydpb_output_delay_length_minus1+1. When sps_max_dec_pic_buffering[max_temporal_layers_minus1] is equal to 0, dpb_output_delay_shall beequal to 0.

The output time derived from the dpb_output_delay of any picture that isoutput from an output timing conforming decoder shall precede the outputtime derived from the dpb_output_delay of all pictures in any subsequentcoded video sequence in decoding order.

The picture output order established by the values of this syntaxelement shall be the same order as established by the values ofPicOrderCntVal.

For pictures that are not output by the “bumping” process because theyprecede, in decoding order, an IDR or BLA picture withno_output_of_prior_pics_flag equal to 1 or inferred to be equal to 1,the output times derived from dpb_output_delay shall be increasing withincreasing value of PicOrderCntVal relative to all pictures within thesame coded video sequence.

num_decoding_units_minus1 plus 1 specifies the number of decoding unitsin the access unit the picture timing SEI message is associated with.The value of num_decoding_units_minus1 shall be in the range of 0 toPicWidthInCtbs*PicHeightInCtbs−1, inclusive.

num_nalus_in_du_minus1 [i] plus 1 specifies the number of NAL units inthe i-th decoding unit of the access unit the picture timing SEI messageis associated with. The value of num_nalus_in_du_minus1 [i] shall be inthe range of 0 to PicWidthInCtbs*PicHeightInCtbs−1, inclusive.

The first decoding unit of the access unit consists of the firstnum_nalus_in_du_minus1 [0]+1 consecutive NAL units in decoding order inthe access unit. The i-th (with i greater than 0) decoding unit of theaccess unit consists of the num_nalus_in_du_minus1 [i]+1 consecutive NALunits immediately following the last NAL unit in the previous decodingunit of the access unit, in decoding order. There shall be at least oneVCL NAL unit in each decoding unit. All non-VCL NAL units associatedwith a VCL NAL unit shall be included in the same decoding unit.

du_cpb_removal_delay [i] specifies how many sub-picture clock ticks towait after removal from the CPB of the first decoding unit in the accessunit associated with the most recent buffering period SEI message in apreceding access unit before removing from the CPB the i-th decodingunit in the access unit associated with the picture timing SEI message.This value is also used to calculate an earliest possible time ofarrival of decoding unit data into the CPB for the HSS. The syntaxelement is a fixed length code whose length in bits is given bycpb_removal_delay_length_minus1+1. The du_cpb_removal_delay [i] is theremainder of a modulo 2^((cpb_removal_delay_length_minus1+1)) counter.

The value of cpb_removal_delay_length_minus1 that determines the length(in bits) of the syntax element du_cpb_removal_delay [i] is the value ofcpb_removal_delay_length_minus1 coded in the sequence parameter set thatis active for the coded picture associated with the picture timing SEImessage, although du_cpb_removal_delay [i] specifies a number ofsub-picture clock ticks relative to the removal time of the firstdecoding unit in the preceding access unit containing a buffering periodSEI message, which may be an access unit of a different coded videosequence.

Some information was contained in the VUI syntax of [2]. The VUIparameters syntax of [2] is shown in FIGS. 3A and 3B. The HRD parameterssyntax of [2] is shown in FIG. 4. The semantics were defined as follows:

sub_pic_cpb_params_present_flag equal to 1 specifies that sub-picturelevel CPB removal delay parameters are present and the CPB may operateat access unit level or sub-picture level.sub_pic_cpb_params_present_flag equal to 0 specifies that sub-picturelevel CPB removal delay parameters are not present and the CPB operatesat access unit level. When sub_pic_cpb_params_present_flag is notpresent, its value is inferred to be equal to 0.

num_units_in_sub_tick is the number of time units of a clock operatingat the frequency time_scale Hz that corresponds to one increment (calleda sub-picture clock tick) of a sub-picture clock tick counter.num_units_in_sub_tick shall be greater than 0. A sub-picture clock tickis the minimum interval of time that can be represented in the codeddata when sub_pic_cpb_params_present_flag is equal to 1.

tiles_fixed_structure_flag equal to 1 indicates that each pictureparameter set that is active in the coded video sequence has the samevalue of the syntax elements num_tile_columns_minus1,num_tile_rows_minus1, uniform_spacing_flag, column_width[i],row_height[i] and loop_filter_across_tiles_enabled_flag, when present.tiles_fixed_structure_flag equal to 0 indicates that tiles syntaxelements in different picture parameter sets may or may not have thesame value. When the tiles_fixed_structure_flag syntax element is notpresent, it is inferred to be equal to 0.

The signaling of tiles_fixed_structure_flag equal to 1 is a guarantee toa decoder that each picture in the coded video sequence has the samenumber of tiles distributed in the same way which might be useful forworkload allocation in the case of multi-threaded decoding.

Filler data of [2] was signaled using filter data RBSP syntax shown inFIG. 5.

The hypothetical reference decoder of [2] used to check bitstream anddecoder conformance was defined as follows:

Two types of bitstreams are subject to HRD conformance checking for thisRecommendation|International Standard. The first such type of bitstream,called Type I bitstream, is a NAL unit stream containing only the VCLNAL units and filler data NAL units for all access units in thebitstream. The second type of bitstream, called a Type II bitstream,contains, in addition to the VCL NAL units and filler data NAL units forall access units in the bitstream, at least one of the following:

-   -   additional non-VCL NAL units other than filler data NAL units,    -   all leading_zero_8 bits, zero_byte,        start_code_prefix_one_3bytes, and trailing_zero_8 bits syntax        elements that form a byte stream from the NAL unit stream.

FIG. 6 shows the types of bitstream conformance points checked by theHRD of [2].

Two types of HRD parameter sets (NAL HRD parameters and VCL HRDparameters) are used. The HRD parameter sets are signaled through videousability information, which is part of the sequence parameter setsyntax structure.

All sequence parameter sets and picture parameter sets referred to inthe VCL NAL units, and corresponding buffering period and picture timingSEI messages shall be conveyed to the HRD, in a timely manner, either inthe bitstream, or by other means.

The specification for “presence” of non-VCL NAL units is also satisfiedwhen those NAL units (or just some of them) are conveyed to decoders (orto the HRD) by other means not specified by thisRecommendation|International Standard. For the purpose of counting bits,only the appropriate bits that are actually present in the bitstream arecounted.

As an example, synchronization of a non-VCL NAL unit, conveyed by meansother than presence in the bitstream, with the NAL units that arepresent in the bitstream, can be achieved by indicating two points inthe bitstream, between which the non-VCL NAL unit would have beenpresent in the bitstream, had the encoder decided to convey it in thebitstream.

When the content of a non-VCL NAL unit is conveyed for the applicationby some means other than presence within the bitstream, therepresentation of the content of the non-VCL NAL unit is not required touse the same syntax specified in this annex.

Note that when HRD information is contained within the bitstream, it ispossible to verify the conformance of a bitstream to the requirements ofthis subclause based solely on information contained in the bitstream.When the HRD information is not present in the bitstream, as is the casefor all “stand-alone” Type I bitstreams, conformance can only beverified when the HRD data is supplied by some other means not specifiedin this Recommendation|International Standard.

The HRD contains a coded picture buffer (CPB), an instantaneous decodingprocess, a decoded picture buffer (DPB), and output cropping as shown inFIG. 7.

The CPB size (number of bits) is CpbSize[SchedSelIdx]. The DPB size(number of picture storage buffers) for temporal layer X issps_max_dec_pic_buffering[X] for each X in the range of 0 tosps_max_temporal_layers_minus1, inclusive.

The variable SubPicCpbPreferredFlag is either specified by externalmeans, or when not specified by external means, set to 0.

The variable SubPicCpbFlag is derived as follows:

SubPicCpbFlag=SubPicCpbPreferredFlag && sub_pic_cpb_params_present_flag

If SubPicCpbFlag is equal to 0, the CPB operates at access unit leveland each decoding unit is an access unit. Otherwise the CPB operates atsub-picture level and each decoding unit is a subset of an access unit.

The HRD operates as follows. Data associated with decoding units thatflow into the CPB according to a specified arrival schedule aredelivered by the HSS. The data associated with each decoding unit areremoved and decoded instantaneously by the instantaneous decodingprocess at CPB removal times. Each decoded picture is placed in the DPB.A decoded picture is removed from the DPB at the later of the DPB outputtime or the time that it becomes no longer needed for inter-predictionreference.

The HRD is initialized as specified by the buffering period SEI. Theremoval timing of decoding units from the CPB and output timing ofdecoded pictures from the DPB are specified in the picture timing SEImessage. All timing information relating to a specific decoding unitshall arrive prior to the CPB removal time of the decoding unit.

The HRD is used to check conformance of bitstreams and decoders.

While conformance is guaranteed under the assumption that allframe-rates and clocks used to generate the bitstream match exactly thevalues signaled in the bitstream, in a real system each of these mayvary from the signaled or specified value.

All the arithmetic is done with real values, so that no rounding errorscan propagate. For example, the number of bits in a CPB just prior to orafter removal of a decoding unit is not necessarily an integer.

The variable t_(c) is derived as follows and is called a clock tick:

t _(c)=num_units_in_tick÷time_scale

The variable t_(c_sub) is derived as follows and is called a sub-pictureclock tick:

t _(c_sub)=num_units_in_sub_tick÷time_scale

The following is specified for expressing the constraints:

-   -   Let access unit n be the n-th access unit in decoding order with        the first access unit being access unit 0.    -   Let picture n be the coded picture or the decoded picture of        access unit n.    -   Let decoding unit m be the m-th decoding unit in decoding order        with the first decoding unit being decoding unit 0.

In [2], the slice header syntax allowed for so-called dependent slices.

FIG. 8 shows the slice header syntax of [2].

Slice header semantics were defined as follows:

dependent_slice_flag equal to 1 specifies that the value of each sliceheader syntax element not present is inferred to be equal to the valueof corresponding slice header syntax element in the preceding slicecontaining the coding tree block for which the coding tree block addressis SliceCtbAddrRS−1. When not present, the value of dependent_slice_flagis inferred to be equal to 0. The value of dependent_slice_flag shall beequal to 0 when SliceCtbAddrRS equal to 0.

slice_address specifies the address in slice granularity resolution inwhich the slice starts. The length of the slice_address syntax elementis (Ceil(Log 2(PicWidthInCtbs*PicHeightInCtbs))+SliceGranularity) bits.

The variable SliceCtbAddrRS, specifying the coding tree block in whichthe slice starts in coding tree block raster scan order, is derived asfollows.

SliceCtbAddrRS=(slice_address>>SliceGranularity)

The variable SliceCbAddrZS, specifying the address of first coding blockin the slice in minimum coding block granularity in z-scan order, isderived as follows.

SliceCbAddrZS=slice_address<<((log2_diff_max_min_coding_block_size−SliceGranularity)<<1)

The slice decoding starts with the largest coding unit possible at theslice starting coordinate.

first_slice_in_pic_flag indicates whether the slice is the first sliceof the picture. If first_slice_in_pic_flag is equal to 1, the variablesSliceCbAddrZS and SliceCtbAddrRS are both set to 0 and the decodingstarts with the first coding tree block in the picture.

pic_parameter_set_id specifies the picture parameter set in use. Thevalue of pic_parameter_set_id shall be in the range of 0 to 255,inclusive.

num_entry_point_offsets specifies the number of entry_point_offset[i]syntax elements in the slice header. Whentiles_or_entropy_coding_sync_idc is equal to 1, the value ofnum_entry_point_offsets shall be in the range of 0 to(num_tile_columns_minus1+1)*(num_tile_rows_minus1+1)−1, inclusive. Whentiles_or_entropy_coding_sync_idc is equal to 2, the value ofnum_entry_point_offsets shall be in the range of 0 to PicHeightInCtbs−1,inclusive. When not present, the value of num_entry_point_offsets isinferred to be equal to 0.

offset_len_minus1 plus 1 specifies the length, in bits, of theentry_point_offset[i] syntax elements.

entry_point_offset[i] specifies the i-th entry point offset, in bytesand shall be represented by offset_len_minus1 plus 1 bits. The codedslice data after the slice header consists of num_entry_point_offsets+1subsets, with subset index values ranging from 0 tonum_entry_point_offsets, inclusive. Subset 0 consists of bytes 0 toentry_point_offset[0]−1, inclusive, of the coded slice data, subset k,with k in the range of 1 to num_entry_point_offsets−1, inclusive,consists of bytes entry_point_offset[k−1] toentry_point_offset[k]+entry_point_offset[k−1]−1, inclusive, of the codedslice data, and the last subset (with subset index equal tonum_entry_point_offsets) consists of the remaining bytes of the codedslice data.

When tiles_or_entropy_coding_sync_idc is equal to 1 andnum_entry_point_offsets is greater than 0, each subset shall contain allcoded bits of exactly one tile, and the number of subsets (i.e., thevalue of num_entry_point_offsets+1) shall be equal to or less than thenumber of tiles in the slice.

When tiles_or_entropy_coding_sync_idc is equal to 1, each slice includeseither a subset of one tile (in which case signaling of entry points isunnecessary) or an integer number of complete tiles.

When tiles_or_entropy_coding_sync_idc is equal to 2 andnum_entry_point_offsets is greater than 0, each subset k with k in therange of 0 to num_entry_point_offsets−1, inclusive, shall contain allcoded bits of exactly one row of coding tree blocks, the last subset(with subset index equal to num_entry_point_offsets) shall contain allcoded bits of the remaining coding blocks included in the slice, whereinthe remaining coding blocks consist of either exactly one row of codingtree blocks or a subset of one row of coding tree blocks, and the numberof subsets (i.e., the value of num_entry_point_offsets+1) shall be equalto the number of rows of coding tree blocks in the slice, wherein asubset of one row of coding tree blocks in the slice is also counted.

When tiles_or_entropy_coding_sync_idc is equal to 2, a slice may includea number of rows of coding tree blocks and a subset of a row of codingtree blocks. For example, if a slice include two and a half rows ofcoding tree blocks, the number of subsets (i.e., the value ofnum_entry_point_offsets+1) shall be equal to 3.

FIG. 9 shows the picture parameter set RBSP syntax of [2], the pictureparameter set RBSP semantics of [2] being defined as:

dependent_slice_enabled_flag equal to 1 specifies the presence of thesyntax element dependent_slice_flag in the slice header for codedpictures referring to the picture parameter set.dependent_slice_enabled_flag equal to 0 specifies the absence of thesyntax element dependent_slice_flag in the slice header for codedpictures referring to the picture parameter set. Whentiles_or_entropy_coding_sync_idc is equal to 3, the value ofdependent_slice_enabled_flag shall be equal to 1.

tiles_or_entropy_coding_sync_idc equal to 0 specifies that there shallbe only one tile in each picture referring to the picture parameter set,there shall be no specific synchronization process for context variablesinvoked before decoding the first coding tree block of a row of codingtree blocks in each picture referring to the picture parameter set, andthe values of cabac_independent_flag and dependent_slice_flag for codedpictures referring to the picture parameter set shall not be both equalto 1.

When cabac_independent_flag and depedent_slice_flag are both equal to 1for a slice, the slice is an entropy slice.]

tiles_or_entropy_coding_sync_idc equal to 1 specifies that there may bemore than one tile in each picture referring to the picture parameterset, there shall be no specific synchronization process for contextvariables invoked before decoding the first coding tree block of a rowof coding tree blocks in each picture referring to the picture parameterset, and the values of cabac_independent_flag and dependent_slice_flagfor coded pictures referring to the picture parameter set shall not beboth equal to 1.

tiles_or_entropy_coding_sync_idc equal to 2 specifies that there shallbe only one tile in each picture referring to the picture parameter set,a specific synchronization process for context variables shall beinvoked before decoding the first coding tree block of a row of codingtree blocks in each picture referring to the picture parameter set and aspecific memorization process for context variables shall be invokedafter decoding two coding tree blocks of a row of coding tree blocks ineach picture referring to the picture parameter set, and the values ofcabac_independent_flag and dependent_slice_flag for coded picturesreferring to the picture parameter set shall not be both equal to 1.

tiles_or_entropy_coding_sync_idc equal to 3 specifies that there shallbe only one tile in each picture referring to the picture parameter set,there shall be no specific synchronization process for context variablesinvoked before decoding the first coding tree block of a row of codingtree blocks in each picture referring to the picture parameter set, andthe values of cabac_independent_flag and dependent_slice_flag for codedpictures referring to the picture parameter set may both be equal to 1.

When dependent_slice_enabled_flag shall be equal to 0,tiles_or_entropy_coding_sync_idc shall not be equal to 3.

It is a requirement of bitstream conformance that the value oftiles_or_entropy_coding_sync_idc shall be the same for all pictureparameter sets that are activated within a coded video sequence.

For each slice referring to the picture parameter set, whentiles_or_entropy_coding_sync_idc is equal to 2 and the first codingblock in the slice is not the first coding block in the first codingtree block of a row of coding tree blocks, the last coding block in theslice shall belong to the same row of coding tree blocks as the firstcoding block in the slice.

num_tile_columns_minus1 plus 1 specifies the number of tile columnspartitioning the picture.

num_tile_rows_minus1 plus 1 specifies the number of tile rowspartitioning the picture.

When num_tile_columns_minus1 is equal to 0, num_tile_rows_minus1 shallnot be equal to 0.

uniform_spacing_flag equal to 1 specifies that column boundaries andlikewise row boundaries are distributed uniformly across the picture.uniform_spacing_flag equal to 0 specifies that column boundaries andlikewise row boundaries are not distributed uniformly across the picturebut signaled explicitly using the syntax elements column_width[i] androw_height[i].

column_width[i] specifies the width of the i-th tile column in units ofcoding tree blocks.

row_height[i] specifies the height of the i-th tile row in units ofcoding tree blocks.

The vector colWidth[i] specifies the width of the i-th tile column inunits of CTBs with the column i ranging from 0 tonum_tile_columns_minus1, inclusive.

The vector CtbAddrRStoTS[ctbAddrRS] specifies the conversation from aCTB address in raster scan order to a CTB address in tile scan orderwith the index ctbAddrRS ranging from 0 to(picHeightInCtbs*picWidthInCtbs)−1, inclusive.

The vector CtbAddrTStoRS[ctbAddrTS] specifies the conversation from aCTB address in tile scan order to a CTB address in raster scan orderwith the index ctbAddrTS ranging from 0 to(picHeightInCtbs*picWidthInCtbs)−1, inclusive.

The vector TileId[ctbAddrTS] specifies the conversation from a CTBaddress in tile scan order to a tile id with ctbAddrTS ranging from 0 to(picHeightInCtbs*picWidthInCtbs)−1, inclusive.

The values of colWidth, CtbAddrRStoTS, CtbAddrTStoRS and TileId arederived by invoking the CTB raster and tile scanning conversationprocess as specified in subclause 6.5.1 with PicHeightInCtbs andPicWidthInCtbs as inputs and the output is assigned to colWidth,CtbAddrRStoTS and TileId.

The values of ColumnWidthInLumaSamples[i], specifying the width of thei-th tile column in units of luma samples, are set equal tocolWidth[i]<<Log 2CtbSize.

The array MinCbAddrZS[x][y], specifying the conversation from a location(x, y) in units of minimum CBs to a minimum CB address in z-scan orderwith x ranging from 0 to picWidthInMinCbs−1, inclusive, and y rangingfrom 0 to picHeightInMinCbs−1, inclusive, is derived by invoking the Zscanning order array initialization process as specified in subclause6.5.2 with Log 2MinCbSize, Log 2CtbSize, PicHeightInCtbs,PicWidthInCtbs, and the vector CtbAddrRStoTS as inputs and the output isassigned to MinCbAddrZS.

loop_filter_across_tiles_enabled_flag equal to 1 specifies that in-loopfiltering operations are performed across tile boundaries.loop_filter_across_tiles_enabled_flag equal to 0 specifies that in-loopfiltering operations are not performed across tile boundaries. Thein-loop filtering operations include the deblocking filter, sampleadaptive offset, and adaptive loop filter operations. When not present,the value of loop_filter_across_tiles_enabled_flag is inferred to beequal to 1.

cabac_independent_flag equal to 1 specifies that CABAC decoding ofcoding blocks in a slice is independent from any state of the previouslydecoded slice. cabac_independent_flag equal to 0 specifies that CABACdecoding of coding blocks in a slice is dependent from the states of thepreviously decoded slice. When not present, the value ofcabac_independent_flag is inferred to be equal to 0.

A derivation process for the availability of a coding block with aminimum coding block address was described as follows:

Inputs to this process are

-   -   a minimum coding block address minCbAddrZS in z-scan order    -   the current minimum coding block address currMinCBAddrZS in        z-scan order

Output of this process is the availability of the coding block withminimum coding block address cbAddrZS in z-scan order cbAvailable.

Note, that the meaning of availability is determined when this processis invoked.

Note, that any coding block, regardless of its size, is associated witha minimum coding block address, which is the address of the coding blockwith the minimum coding block size in z-scan order.

-   -   If one or more of the following conditions are true, cbAvailable        is set to FALSE.        -   minCbAddrZS is less than 0        -   minCbAddrZS is greater than currMinCBAddrZS        -   the coding block with minimum coding block address            minCbAddrZS belongs to a different slice than the coding            block with the current minimum coding block address            currMinCBAddrZS and the dependent slice flag of the slice            containing the coding block with the current minimum coding            block address currMinCBAddrZS is equal to 0.        -   the coding block with minimum coding block address            minCbAddrZS is contained in a different tile than the coding            block with the current minimum coding block address            currMinCBAddrZS.    -   Otherwise, cbAvailable is set to TRUE.

The CABAC parsing process for slice data of [2] was as follows:

This process is invoked when parsing syntax elements with descriptorae(v).

Inputs to this process are a request for a value of a syntax element andvalues of prior parsed syntax elements.

Output of this process is the value of the syntax element.

When starting the parsing of the slice data of a slice, theinitialization process of the CABAC parsing process is invoked.

The minimum coding block address of the coding tree block containing thespatial neighbor block T (FIG. 10A), ctbMinCbAddrT, is derived using thelocation (x0, y0) of the top-left luma sample of the current coding treeblock as follows.

x=x0+2<<Log 2CtbSize−1

y=y0−1

ctbMinCbAddrT=MinCbAddrZS[x>>Log 2MinCbSize][y>>Log 2MinCbSize]

The variable availableFlagT is obtained by invoking the coding blockavailability derivation process with ctbMinCbAddrT as input.

When starting the parsing of a coding tree, the following ordered stepsapply.

The arithmetic decoding engine is initialized as follows.

-   -   If CtbAddrRS is equal to slice_address, dependent_slice_flag is        equal to 1 and entropy_coding_reset_flag is equal to 0, the        following applies.        -   The synchronization process of the CABAC parsing process is            invoked with TableStateIdxDS and TableMPSValDS as input.        -   The decoding process for binary decisions before termination            is invoked, followed by the initialization process for the            arithmetic decoding.    -   Otherwise if tiles_or_entropy_coding_sync_idc is equal to 2, and        CtbAddrRS % PicWidthInCtbs is equal to 0, the following applies.        -   When availableFlagT is equal to 1, the synchronization            process of the CABAC parsing process is invoked with            TableStateIdxWPP and TableMPSValWPP as input        -   The decoding process for binary decisions before termination            is invoked, followed by the initialization process for the            arithmetic decoding engine.

When cabac_independent_flag is equal to 0 and dependent_slice_flag isequal to 1, or when tiles_or_entropy_coding_sync_idc is equal to 2, thememorization process is applied as follows.

-   -   When tiles_or_entropy_coding_sync_idc is equal to 2 and        CtbAddrRS % PicWidthInCtbs is equal to 2, the memorization        process of the CABAC parsing process is invoked with        TableStateIdxWPP and TableMPSValWPP as output.    -   When cabac_independent_flag is equal to 0, dependent_slice_flag        is equal to 1, and end_of_slice_flag is equal to 1, the        memorization process of the CABAC parsing process is invoked        with TableStateIdxDS and TableMPSValDS as output.

The parsing of syntax elements proceeds as follows:

For each requested value of a syntax element a binarization is derived.

The binarization for the syntax element and the sequence of parsed binsdetermines the decoding process flow.

For each bin of the binarization of the syntax element, which is indexedby the variable binIdx, a context index ctxIdx is derived.

For each ctxIdx the arithmetic decoding process is invoked.

The resulting sequence (b₀ . . . b_(binIdx)) of parsed bins is comparedto the set of bin strings given by the binarization process afterdecoding of each bin. When the sequence matches a bin string in thegiven set, the corresponding value is assigned to the syntax element.

In case the request for a value of a syntax element is processed for thesyntax element pcm-flag and the decoded value of pcm_flag is equal to 1,the decoding engine is initialized after the decoding of anypcm_alignment_zero_bit, num_subsequent_pcm, and all pcm_sample_luma andpcm_sample_chroma data.

In the design framework described so far the following problem occurred.

The timing of the decoding units need to be known before coding andsending the data in a low delay scenario, where NAL units will alreadybe sent out by the encoder, while the encoder is still coding parts ofthe picture, i.e. other sub-picture decoding units. This is, because theNAL unit order in an access unit only allows SEI messages to precede theVCL (Video Coding NAL units) in an access unit, but in such a low delayscenario, the non-VCL NAL units need to be already on the wire, i.e.sent out, if the encoder starts encoding the decoding units.

FIG. 10B illustrates the structure of an access unit as defined in [2].[2] did not yet specify end of sequence or stream, so their presence inthe access unit was tentative.

Furthermore, the number of NAL units associated with a sub-picture alsoneeds to be known beforehand in a low delay scenario, as the picturetiming SEI message contains this information and has to be compiled andsend out before the encoder starts to encode the actual picture. Anapplication designer reluctant to insert filler data NAL units, withpotentially no filler data to comply with the NAL unit number, assignaled per decoding unit in the picture timing SEI, needs means tosignal this information on a sub-picture level. The same holds forsub-picture timing, which is currently fixed at the being of an accessunit by the parameters given in the timing SEI message.

Further shortcomings of the draft specification [2] include numeroussignaling of sub-picture level, which is used for specific applications,such as ROI signaling or tile dimensions signaling.

The above outlined problems are not specific to the HEVC standard.Rather, this problem also occurs in connection with other video codecsas well. FIG. 11 shows, more generally, a video transmission scenerywhere a pair encoder 10 and decoder 12 are connected via a network 14 inorder to transmit a video 16 from encoder 10 to decoder 12 at shortend-to-end delay. The problem already outlined above is the following.The encoder 10 encodes the sequence of frames 18 of the video 16 inaccordance with a certain decoding order which substantially, but notnecessarily, follows the reproduction order 20 of frames 18, and withineach frame 18 travels through the frame area of frames 18 in somedefined manner, such as for example in a raster scan manner with orwithout tile-sectioning of frames 18. The decoding order controls theavailability of information for coding techniques used by encoder 10such as, for example, prediction and/or entropy coding, i.e. theavailability of information relating to spatially and/or temporallyneighboring portions of video 16 available to serve as a basis forprediction or context selection. Even though encoder 10 might be able touse parallel processing in order to encode the frames 18 of video 16,encoder 10 needs some time to encode a certain frame 18, such as thecurrent frame. FIG. 11, for example, illustrates a time instant whereencoder 10 has already finished encoding portion 18 a of a current frame18, while another portion 18 b of current frame 18 has not yet beenencoded. As encoder 10 has not yet encoded portion 18 b, encoder 10 maynot forecast how the available bitrate for encoding current frame 18should be distributed spatially over current frame 18 to achieve anoptimum in terms of, for example, rate/distortion sense. Accordingly,encoder 10 merely has two choices: either encoder 10 estimates anearly-optimum distribution of the available bitrate for current frame18 onto the slices into which current frame 18 is spatially subdividedin advance, accordingly accepting that the estimation may be wrong, orencoder 10 finalizes encoding current frame 18 prior to transmitting thepackets containing the slices from encoder 10 to decoder 12. In anycase, in order to be able to take advantage of any transmission of slicepackets of current coded frame 18 prior to the finalization of itsencoding, network 14 should be informed of the bitrates associated witheach such slice packet in the form of coded picture buffer retrievaltimes. However, as indicated above, although encoder 10 is, inaccordance with the current version of HEVC, able to vary the bitratedistributed over frames 18 by use of defining decoder buffer retrievaltimes for sub-picture areas individually, encoder 10 needs to transmitor send out such information via network 14 to decoder 12 at thebeginning of each access unit collecting all data relating to currentframe 18, thereby urging encoder 10 to choose among the just outlinedtwo alternatives, one leading to lower delay but worse rate/distortion,the other leading to optimum rate/distortion, however at increasedend-to-end delay.

Thus, so far there is no video codec enabling the achievement of such alow delay that the encoder would be enabled to start transmittingpackets relating to portions 18 a of the current frame prior to encodinga remaining portion 18 b of the current frame, the decoder being able toexploit this intermediate transmission of packets relating topreliminary portions 18 a by way of the network 16, which obeys thedecoding buffer retrieval timing conveyed within the video data streamsent from encoder 12 to decoder 14. Applications which would, forexample, take advantage of such low delay exemplarily encompassindustrial applications such as, for example, work piece or fabricationsurveillance for automation or inspection purposes or the like. Untilnow, there is also no satisfactory solution for informing the decodingside on the packets' association to tiles into which a current frame isstructured, and interesting regions (region of interest) of a currentframe so that intermediate network entities within network 16 areenabled to gather such information from the data stream without havingto deeply inspect the inside of the packets, i.e. the slices syntax.

SUMMARY

According to an embodiment, a video data stream may have: video contentencoded therein in units of sub-portions of pictures of the videocontent, each sub-portion being respectively encoded into one or morepayload packets of a sequence of packets of the video data stream, thesequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets relating to arespective picture of the video content, wherein the sequence of packetshas interspersed thereinto timing control packets so that the timingcontrol packets subdivide the access units into decoding units so thatat least some access units are subdivided into two or more decodingunits, with each timing control packet signaling a decoder bufferretrieval time for a decoding unit, the payload packets of which followthe respective timing control packet in the sequence of packets.

According to another embodiment, an encoder for encoding into a videodata stream video content in units of sub-portions of pictures of thevideo content, with respectively encoding each sub-portion into one ormore payload packets of a sequence of packets of the video data streamso that the sequence of packets is divided into a sequence of accessunits and each access unit collects the payload packets relating to arespective picture of the video content, may be configured tointersperse into the sequence of packets timing control packets so thatthe timing control packets subdivide the access units into decodingunits so that at least some access units are subdivided into two or moredecoding units, with each timing control packet signaling a decoderbuffer retrieval time for a decoding unit, the payload packets of whichfollow the respective timing control packet in the sequence of packets.

According to another embodiment, a method for encoding into a video datastream video content in units of sub-portions of pictures of the videocontent, with respectively encoding each sub-portion into one or morepayload packets of a sequence of packets of the video data stream sothat the sequence of packets is divided into a sequence of access unitsand each access unit collects the payload packets relating to arespective picture of the video content, may have the steps of:interspersing into the sequence of packets timing control packets sothat the timing control packets subdivide the access units into decodingunits so that at least some access units are subdivided into two or moredecoding units, with each timing control packet signaling a decoderbuffer retrieval time for a decoding unit, the payload packets of whichfollow the respective timing control packet in the sequence of packets.

According to another embodiment, a decoder for decoding a video datastream having video content encoded therein in units of sub-portions ofpictures of the video content, each sub-portion being respectivelyencoded into one or more payload packets of a sequence of packets of thevideo data stream, the sequence of packets being divided into a sequenceof access units so that each access unit collects the payload packetsrelating to a respective picture of the video content, may have a bufferfor buffering the video data stream or a reconstruction of the videocontent acquired therefrom by the decoding of the video data stream andbe configured to look for timing control packets interspersed into thesequence of packets, subdivide the access units into decoding units atthe timing control packets so that at least some access units aresubdivided into two or more decoding units, and empty the buffer inunits of the decoding units.

According to another embodiment, a method for decoding a video datastream having video content encoded therein in units of sub-portions ofpictures of the video content, each sub-portion being respectivelyencoded into one or more payload packets of a sequence of packets of thevideo data stream, the sequence of packets being divided into a sequenceof access units so that each access unit collects the payload packetsrelating to a respective picture of the video content, may have thesteps of: using a buffer for buffering the video data stream or areconstruction of the video content acquired therefrom by the decodingof the video data stream and including looking for timing controlpackets interspersed into the sequence of packets, subdividing theaccess units into decoding units at the timing control packets so thatat least some access units are subdivided into two or more decodingunits, and emptying the buffer in units of the decoding units.

According to another embodiment, a network entity for transmitting avideo data stream may have video content encoded therein in units ofsub-portions of pictures of the video content, each sub-portion beingrespectively encoded into one or more payload packets of a sequence ofpackets of the video data stream, the sequence of packets being dividedinto a sequence of access units so that each access unit collects thepayload packets relating to a respective picture of the video content,the decoder being configured to look for timing control packetsinterspersed into the sequence of packets, subdivide the access unitsinto decoding units at the timing control packets so that at least someaccess units are subdivided into two or more decoding units, derive fromeach timing control packet a decoder buffer retrieval time for adecoding unit, the payload packets of which follow the respective timingcontrol packet in the sequence of packets, and perform the transmissionof the video data stream dependent on the decoder buffer retrieval timesfor the decoding units.

According to another embodiment, a method for transmitting a video datastream having video content encoded therein in units of sub-portions ofpictures of the video content, each sub-portion being respectivelyencoded into one or more payload packets of a sequence of packets of thevideo data stream, the sequence of packets being divided into a sequenceof access units so that each access unit collects the payload packetsrelating to a respective picture of the video content, may have thesteps of: looking for timing control packets interspersed into thesequence of packets, subdivide the access units into decoding units atthe timing control packets so that at least some access units aresubdivided into two or more decoding units, deriving from each timingcontrol packet a decoder buffer retrieval time for a decoding unit, thepayload packets of which follow the respective timing control packet inthe sequence of packets, and performing the transmission of the videodata stream dependent on the decoder buffer retrieval times for thedecoding units.

Another embodiment may have a video data stream having video contentencoded therein, using predictive and entropy coding, in units of slicesinto which pictures of the video content are spatially sub-divided,using a coding order among the slices, with restricting predictions ofthe predictive coding and/or entropy coding to the inner of tiles intowhich the pictures of the video content are spatially sub-divided,wherein the sequence of the slices are packetized into payload packetsof a sequence of packets of the video data stream in the coding order,the sequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets having packetizedthereinto slices relating to a respective picture of the video content,wherein the sequence of packets has tile identification packetsinterspersed thereinto between payload packets of one access unit,identifying one or more tiles overlaid by any slice packetized into oneor more payload packets immediately following the respective tileidentification packet in the sequence of packets.

According to another embodiment, a network entity may be configured toreceive a video data stream according to claim 15 and identify, based onthe tile identification packets, tiles which are overlaid by slicespacketized into one or more payload packets immediately following therespective tile identification packet in the sequence of packets.

According to another embodiment, a method may have the steps of:receiving a video data stream according to claim 15 and identifying,based on the tile identification packets, tiles which are overlaid byslices packetized into one or more payload packets immediately followingthe respective tile identification packet in the sequence of packets.

Another embodiment may have a video data stream having video contentencoded therein in units of sub-portions of pictures of the videocontent, each sub-portion being respectively encoded into one or morepayload packets of a sequence of packets of the video data stream, thesequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets relating to arespective picture of the video content, wherein at least some accessunits have the sequence of packets has interspersed thereinto ROIpackets so that the timing control packets subdivide the access unitsinto decoding units so that at least some access units have ROI packetsinterspersed between payload packets relating to the picture of therespective access unit, with each ROI packet relating to one or morefollowing payload packets in the sequence of packets, following therespective ROI packet, and identifying as to whether the sub-portionsencoded into any of the one or more payload packets to which therespective ROI packet relates, overlay a region of interest of the videocontent.

According to another embodiment, a network entity may be configured toreceive a video data stream according to claim 24 and identify, based onthe ROI packets, the ROI of the video content.

According to another embodiment, a method may have the steps of:receiving a video data stream according to claim 24 and identifying,based on the ROI packets, the ROI of the video content.

Another embodiment may have a computer-program having a program code forperforming, when running on a computer, a method according to claims 9,12, 14, 23, and 33.

One idea on which the present application is based, is that decoderretrieval timing information, ROI information and tile identificationinformation should be conveyed within a video data stream at a levelwhich allows for an easy access by network entities such as MANEs ordecoders and that, in order to reach such a level, information of suchtypes should be conveyed within a video data stream by way of packetsinterspersed into packets of access units of a video data stream. Inaccordance with an embodiment, the interspersed packets are of aremovable packet type, i.e. the removal of these interspersed packetsmaintains the decoder's ability to completely recover the video contentconveyed via the video data stream.

In accordance with an aspect of the present application, the achievementof low end-to-end delay is rendered more effective by using theinterspersed packets in order to convey information on decoder bufferretrieval times for decoding units formed by payload packets whichfollow the respective timing control packet in the video data streamwithin the current access unit. By this measure, the encoder is enabledto determine the decoder buffer retrieval times on the fly duringencoding a current frame, thereby being able to, while encoding acurrent frame, continuously determine the bitrate actually spent for theportion of the current frame having already been encoded into payloadpackets and transmitted, or sent out, prefixed with timing controlpackets, on the one hand, and accordingly adapt the distribution of theremaining bitrate available for the current frame over the remainingportion of the current frame not yet having been encoded. By thismeasure, the bitrate available is effectively exploited and the delay isnevertheless kept shorter as the encoder does not need to wait to finishencoding the current frame completely.

In accordance with a further aspect of the present application, packetsinterspersed into a payload packet of an access unit are exploited toconvey information on a region of interest, thereby enabling as outlinedabove an easy access of this information by network entities as they donot have to inspect the intermediate payload packets. Further, theencoder is still free to determine the packets belonging to the ROIduring encoding a current frame on the fly without having to determinethe current frame's subdivision into sub-portions and respective payloadpackets in advance. Moreover, in accordance with the embodimentaccording to which the interspersed packets are of a removable packettype, the ROI information may be disregarded by recipients of the videodata stream not interested in the ROI information, or not able toprocess same.

Similar thoughts are exploited in the present application in accordancewith another aspect according to which the interspersed packets conveyinformation on which tile certain packets within an access unit belongto.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIGS. 1 to 10B show a current status of the HEVC with FIG. 1 showing abuffering period SEI message syntax, FIG. 2 showing a picture timing SEImessage syntax, FIGS. 3A and 3B showing a VUI parameter syntax, FIG. 4showing an HRD parameter syntax, FIG. 5 showing filler data RBSP syntax,FIG. 6 showing a structure of byte streams and NAL unit streams for HRDconformance checks, FIG. 7 showing an HRD buffer model, FIG. 8 showing aslice header syntax, FIG. 9 showing a picture parameter set RBSP syntax,FIG. 10A showing a schematic illustrating a spatially neighboring codetree block T possibly used to invoke the coding tree block availabilityderivation process relative to the current coding tree block and FIG.10B showing a definition of a structure of an access unit;

FIG. 11 schematically shows a pair of encoder and decoder connected viaa network for illustrating problems occurring in video data streamtransmission;

FIG. 12 shows a schematic block diagram of an encoder in accordance withan embodiment using timing control packets;

FIG. 13 shows a flow diagram illustrating a mode of operation of theencoder of FIG. 12 in accordance with an embodiment;

FIG. 14 shows a block diagram of an embodiment of a decoder so as toexplain its functionality in connection with a video data streamgenerated by an encoder according to FIG. 12;

FIG. 15 shows a schematic block diagram illustrating an encoder, networkentity and video data stream in accordance with a further embodimentusing ROI packets;

FIG. 16 shows a schematic block diagram illustrating an encoder, networkentity and video data stream in accordance with a further embodimentusing tile identification packets;

FIG. 17 shows a structure of an access unit according to an embodiment.The dashed line reflects the case of a non-mandatory slice prefix NALunit;

FIG. 18 shows the use of tiles in region of interest signaling;

FIG. 19 shows the first simple syntax/version 1;

FIG. 20 shows the extended syntax/version 2 including tile_id signaling,decoding unit start identifier, slice prefix ID and slice header dataapart from the SEI message concept;

FIG. 21 shows NAL unit type code and NAL unit type classes;

FIG. 22 shows a possible syntax for a slice header, where certain syntaxelements present in the slice header according to the current versionare shifted to a lower hierarchy syntax element, referred to asslice_header_data( );

FIGS. 23A to 23C show a table where all syntax elements removed from theslice header are signaled through the syntax element slice header data;

FIG. 24 shows a supplemental enhancement information message syntax;

FIGS. 25A and 25B show an adapted SEI payload syntax in order tointroduce new slice or sub-picture SEI message types;

FIG. 26 shows an example for a sub-picture buffering SEI message;

FIG. 27 shows an example for a sub-picture timing SEI message;

FIG. 28 shows how the sub-picture slice info SEI message may look like;

FIG. 29 shows an example for a sub-picture tile info SEI message;

FIG. 30 shows a syntax example for a sub-picture tile dimension info SEImessage;

FIG. 31 shows a first variant of a syntax example for a region ofinterest SEI message where each ROI is signaled in an individual SEImessage;

FIG. 32 shows a second variant of a syntax example for a region ofinterest SEI message where all ROIs are signaled in a single SEImessage;

FIG. 33 shows a possible syntax for a timing control packet inaccordance with a further embodiment;

FIG. 34 shows a possible syntax for a tile identification packet inaccordance with an embodiment;

FIGS. 35 to 38 show possible subdivisions of a picture in accordancewith different subdivision settings in accordance with an embodiment;and

FIG. 39 shows an example of a portion out of a video data stream inaccordance with an embodiment using timing control packets beinginterspersed between the payload packets of an access unit.

DETAILED DESCRIPTION OF THE INVENTION

With regard to FIG. 12, an encoder 10 in accordance with an embodimentof the present application and its mode of operation is described. Theencoder 10 is configured to encode video content 16 into a video datastream 22. The encoder is configured to do this in units of sub-portionsof the frames/pictures 18 of the video content 16, wherein thesub-portions may, for example, be slices 24 into which the pictures 18are partitioned, or some other spatial segments such as, for example,tiles 26 or WPP substreams 28, all of which are illustrated in FIG. 12merely for the sake of illustrative purposes rather than suggesting thatencoder 10 needs to be able to support tile or WPP parallel processing,for example, or that the sub-portions need to be slices.

In encoding the video content 16 in units of the sub-portions 24, theencoder 10 may obey a decoding order—or coding order—defined among thesub-portions 24, which for example traverses pictures 18 of video 16 inaccordance with a picture decoding order which, for example, does notnecessarily coincide with the reproduction order 20 defined amongpictures 18, and traverses within each picture 18 blocks into whichpictures 18 are partitioned, in accordance with a raster scan order,with the sub-portions 24 representing continuous runs of such blocksalong the decoding order. In particular, encoder 10 may be configured toobey this decoding order in determining the availability of spatiallyand/or temporally neighboring portions of portions currently to beencoded in order to use attributes describing such neighboring portionsin predictive coding and/or entropy coding such as, for example, todetermine a prediction and/or an entropy context: Merely previouslyvisited (coded/decoded) portions of the video are available. Otherwise,just-mentioned attributes are set to default values or some othersubstitute measures are taken.

On the other hand, encoder 10 does not need to serially encodesub-portions 24 along the decoding order. Rather, encoder 10 may useparallel processing to speed-up the encoding process, or to be able toperform a more complex encoding in real time. Likewise, encoder 10 mayor may not be configured to transmit or send-out the data encoding thesub-portions along the decoding order. For example, encoder 10 mayoutput/transmit the encoded data at some other order such as, forexample, in accordance with the order at which the encoding of thesub-portions is finalized by encoder 10 which may, due to the parallelprocessing, for example, deviate from the decoding order just-mentioned.

In order to render the encoded versions of sub-portions 24 suitable fortransmission over a network, encoder 10 encodes each sub-portion 24 intoone or more payload packets of a sequences of packets of video datastream 22. In case of the sub-portions 24 being slices, encoder 10 may,for example, be configured to put each slice data, i.e. each encodedslice, into one payload packet, such as an NAL unit. This packetizationmay serve to render the video data stream 22 appropriate fortransmission via a network. Accordingly, packets may represent thesmallest units at which the video data stream 22 may take place, i.e.the smallest units which may be individually sent-out by encoder 10 fortransmission via a network to a recipient.

Besides payload packets and the timing control packets interspersedtherebetween and discussed hereinafter, other packets, i.e. packets ofother type may exist as well, such as fill data packets, picture orsequence parameter set packets for conveying infrequently changingsyntax elements or EOF (end of file) or AUE (access unit end) packets orthe like.

The encoder performs the encoding into the payload packets such that thesequence of packets is divided into a sequence of access units 30 andeach access unit collects the payload packets 32 relating to one picture18 of the video content 16. That is, the sequence 34 of packets formingvideo data stream 22 is subdivided into non-overlapping portions, calledaccess units 30, each being associated with a respective one of thepictures 18. The sequence of access units 30 may follow the decodingorder of the pictures 18 which the access units 30 relate to. FIG. 12illustrates, for example, that the access unit 30 arranged at the mid ofthe portion of data stream 22, illustrated, comprises one payload packet32 per sub-portion 24 into which picture 18 is subdivided. That is, eachpayload packet 32 carries a corresponding sub-portion 24. The encoder 10is configured to intersperse into the sequence 34 of packets timingcontrol packets 36 so that the timing control packets subdivide theaccess units 30 into decoding units 38 so that at least some accessunits 30, such as the middle one shown in FIG. 12, are subdivided intotwo or more decoding units 38, each timing control packet signaling adecoder buffer retrieval time for a decoding unit 38, the payloadpackets 32 of which follow the respective timing control packet in thesequence 34 of packets. In other words, encoder 10 prefixes subsequencesof the sequence of payload packets 32 within one access unit 30 with arespective timing control packet 36 signaling for the respectivesubsequence of payload packets prefixed by the respective timing controlpacket 36 and forming a decoding unit 38, a decoder buffer retrievaltime. FIG. 12, for example, illustrates the case where every secondpacket 32 represents the first payload packet of a decoding unit 38 ofaccess unit 30. As illustrated in FIG. 12, the amount of data or bitratespent for each decoding unit 38 varies and the decoder buffer retrievaltimes may correlate with this bitrate variation among the decoding units38 in that the decoder buffer retrieval time of a decoding unit 38 mayfollow the decoder buffer retrieval time signaled by the timing controlpacket 36 of the immediately preceding decoding unit 38 plus a timeinterval corresponding to the bitrate spent for this immediatelypreceding decoding unit 38.

That is, the encoder 10 may operate as shown in FIG. 13. In particular,as mentioned above encoder 10 may, in a step 40, subject a currentsub-portion 24 of a current picture 18 to an encoding. As alreadymentioned, encoder 10 may sequentially cycle through the sub-portions 24in the aforementioned decoding order as illustrated by arrow 42, orencoder 10 may use some parallel processing such as WPP and/or tileprocessing in order to concurrently encode several “currentsub-portions” 24. Irrespective of using parallel processing or not,encoder 10 forms a decoding unit out of one or several of thesub-portions just encoded in step 40 and proceeds with step 44, wherethe encoder 10 sets a decoder buffer retrieval time for this decodingunit and transmits this decoding unit prefixed with a time controlpacket signaling the just set decoder buffer retrieval time for thisdecoding unit. For example, the encoder 10 may determine the decoderbuffer retrieval time in step 44 on the basis of the bitrate spent forencoding the sub-portions having been encoded into the payload packetsforming the current decoding unit including, for example, all furtherintermediate packets within this decoding unit, if any, i.e. the“prefixed packets”.

Then, in step 46, the encoder 10 may adapt the available bitrate on thebasis of the bitrate having been spent for the decoding unit just havingbeen transmitted in step 44. If, for example, the picture content withinthe decoding unit just-transmitted in step 44 was quite complex in termsof compression rate, then encoder 10 may reduce the available bitratefor the next decoding unit so as to obey some externally set targetbitrate having been determined, for example, on the basis of a currentbandwidth situation faced in connection with the network transmittingthe video data stream 22. Steps 40 to 46 are then repeated. By thismeasure, pictures 18 are encoded and transmitted, i.e. sent out, inunits of decoding units, each being prefixed by a corresponding timingcontrol packet.

In other words, the encoder 10, during encoding a current picture 18 ofthe video content 16, encodes 40 a current sub-portion 24 of the currentpicture 18 into a current payload packet 32 of a current decoding unit38, transmits 44, within the data stream, the current decoding unit 38prefixed with a current timing control packet 36 with setting a decoderbuffer retrieval time signaled by the current timing control packet(36), at a first time instant, and encodes 44, by looping back from step46 to 40, a further sub-portion 24 of the current picture 18 at a secondtime instant second time visiting step 40—, later than the first timeinstant—first time visiting step 44.

As the encoder is able to send-out a decoding unit prior to the encodingof a remainder of the current picture to which this decoding unitbelongs, encoder 10 is able to lower the end-to-end delay. On the otherhand, encoder 10 does not need to waste available bitrate, as theencoder 10 is able to react to the specific nature of the content of thecurrent picture and its spatial distribution of complexity.

On the other hand, intermediate network entities, responsible fortransmitting the video data stream 22 further from the encoder to thedecoder, are able to use the timing control packets 36 so as toguarantee that any decoder receiving the video data stream 22 receivesthe decoding units in time so as to be able to gain advantage of thedecoding unit-wise encoding and transmission by encoder 10. See, forexample, FIG. 14 showing an example for a decoder for decoding the videodata stream 22. Decoder 12 receives the video data stream 22 at a codedpicture buffer CPB 48 by way of a network via which encoder 10transmitted the video data stream 22 to decoder 12. In particular, asnetwork 14 is assumed to be able to support the low delay application,network 10 inspects the decoder buffer retrieval times so as to forwardthe sequence 34 of packets of video data stream 22 to the coded picturebuffer 48 of decoder 12 so that each decoding unit is present within thecoded picture buffer 48 prior to the decoder buffer retrieval timesignaled by the timing control packet prefixing the respective decodingunit. By this measure, the decoder is able to, without stalling, i.e.without running out of available payload packets within coded picturebuffer 48, use the decoder buffer retrieval times in the timing controlpackets so as to empty the decoder's coded picture buffer 48 in units ofthe decoding units rather than complete access units. FIG. 14, forexample, shows for illustrative purposes, a processing unit 50 as beingconnected to an output of coded picture buffer 48, the input of whichreceives the video data stream 22. Similar to encoder 10, decoder 12 maybe able to perform parallel processing such as, for example, using tileparallel processing/decoding and/or WPP parallel processing/decoding.

As will be outlined in more detail below, the decoder buffer retrievaltimes mentioned so far do not necessarily pertain to retrieval timesconcerning the coded picture buffer 48 of decoder 12. Rather, the timingcontrol packets may additionally, or alternatively, steer the retrievalof already decoded picture data of a corresponding decoded picturebuffer of a decoder 12. FIG. 14 shows, for example, decoder 12 ascomprising a decoder picture buffer in which the decoded version of thevideo content, as obtained by processing unit 50 by decoding video datastream 22, is buffered, i.e. stored and output, in units of decodedversions of decoding units. Decoder's decoded picture buffer 22 thus maybe connected between decoder's 12 output and the output of processingunit 50. By having the ability to set the retrieval times for outputtingdecoded versions of decoding units from decoded picture buffer 52, theencoder 10 is given the opportunity to, on the fly, i.e. during encodinga current picture, control the reproduction, or end-to-end, delay of thereproduction of the video content at decoding side even at a granularitysmaller than the picture rate or frame rate. Obviously, over-segmentingeach picture 18 into a huge amount of sub-portions 24 at the encodingside would negatively affect the bitrate for transmitting the video datastream 22, although on the other hand, the end-to-end delay may beminimized since the time needed to encode and transmit and decode andoutput such a decoding unit would be minimized. On the other hand,increasing the size of the sub-portions 24 increases the end-to-enddelay. Accordingly, a compromise has to be found. Using the justmentioned decoder buffer retrieval times so as to steer the outputtiming of decoded versions of sub-portions 24 in units of decodingunits, enables the encoder 10 or some other unit at the encoding side toadapt this compromise spatially over the current picture's content. Bythis measure, it would be possible to control the end-to-end delay insuch a way, that same varies spatially across the current picturescontent.

In implementing the above outlined embodiments, it is possible to use,as the timing control packets, packets of a removable packet type.Packets of a removable packet type are not necessary in order to recoverthe video content at the decoding side. In the following, such packetsare called SEI packets. Further packets of a removable packet type mayexist as well, that is, removable packets of another type such as, iftransmitted in-stream, redundancy packets. As another alternative,timing control packets may be packets of a certain removable packettype, additionally carrying, however, a certain SEI packet type field.For example, timing control packets may be SEI packets with each SEIpacket carrying one or several SEI messages, and only those SEI packetswhich comprise an SEI message of a certain type form the aforementionedtiming control packets.

Thus, the embodiment described so far with respect to FIGS. 12 to 14 is,in accordance with a further embodiment, applied onto the HEVC standard,thereby forming a possible concept for rendering HEVC more effective inachieving lower end-to-end delay. In doing so, the above mentionedpackets are formed by NAL units and the aforementioned payload packetsare the VCL NAL units of an NAL unit stream with slices forming theabove mentioned sub-portions.

Before such a description of a more detailed embodiment, however,further embodiments are described which coincide with the above outlinedembodiments in that interspersed packets are used in order to convey, inan efficient manner, information describing the video data stream, butthe sort of information differs from the above embodiments where thetiming control packets conveyed decoder buffer retrieval timinginformation. In the embodiments described further below, the kind ofinformation transferred via interspersed packets interspersed into thepayload packets belonging to an access unit, relate to region ofinterest (ROI) information and/or tile identification information. Theembodiments described further below may or may not be combined with theembodiments described with respect to FIGS. 12 to 14.

FIG. 15 shows an encoder 10 which operates similar to the one explainedabove with respect to FIG. 12, except for the interspersing of timingcontrol packets and the functionality described above with respect toFIG. 13, which is optional for encoder 10 of FIG. 15. However, encoder10 of FIG. 15 is configured to encode video content 16 into a video datastream 22 in units of sub-portions 24 of the pictures 18 of the videocontent 16 just as it was explained above with respect to FIG. 11. Inencoding the video content 16, encoder 10 is interested in conveying,along with the video data stream 22, information on a region of interestROI 60 to the decoding side. The ROI 60 is a spatial subarea of acurrent picture 18, which the decoder should, for example, pay specialattention to. The spatial position of the ROI 60 may be input to encoder10 from outside, as illustrated by a dashed line 62, such as by userinput, or may be determined automatically by encoder 10 or by some otherentity, on the fly during the encoding of current picture 18. In eithercase, encoder 10 faces the following problem: the indication of thelocation of the ROI 60 is in principle no problem for encoder 10. To dothis, the encoder 10 may easily indicate the location of the ROI 60within the data stream 22. However, in order to render this informationeasily accessible, encoder 10 of FIG. 15 uses the interspersing of ROIpackets between the payload packets of the access units so that encoder10 is free to, on an online basis, choose the segmentation of thecurrent picture 18 into sub-portions 24 and/or the number of payloadpackets into which the sub-portions 24 are packetized, spatially outsideand spatially inside the ROI 60. Using the interspersed ROI packets, anynetwork entity may easily identify payload packets which belong to theROI. On the other hand, in case of using removable packet type for theseROI packets, same may easily be disregarded by any network entity.

FIG. 15 shows an example for interspersing ROI packets 64 between thepayload packets 32 of an access unit 30. The ROI packet 64 indicateswhere within the sequence 34 of packets of the video data stream 22encoded data is contained which relates to, i.e. encodes, the ROI 60.How ROI packet 64 indicates the location of ROI 60 may be implemented inmanifold ways. For example, the pure existence/occurrence of an ROIpacket 64 may indicate the incorporation of encoded data relating to theROI 60 within one or more of the following payload packets 32, followingin the sequential order of sequence 34, i.e. belonging to the prefixedpayload packets. Alternatively, a syntax element inside the ROI packet64 may indicate whether one or more following payload packets 32 pertainto, i.e. at least partially encode, the ROI 60 or not. The high numberof variance also stems from possible variations regarding the “scope” ofthe respective ROI packet 64, i.e. the number of prefixed payloadpackets prefixed by one ROI packet 64. For example, the indication ofincorporation or non-incorporation of any encoded data relating to theROI 60 within one ROI packet, may relate to all payload packets 32following in the sequential order of sequence 34 up to the occurrence ofthe next ROI packet 64, or may merely relate to the immediatelyfollowing payload packet 32, i.e. the payload packet 32 immediatelyfollowing the respective ROI packet 64 in the sequential order ofsequence 34. In FIG. 15, a graph 66 exemplarily illustrates a case wherethe ROI packets 64 indicate an ROI relevance, i.e. the incorporation ofany encoded data relating to the ROI 60, or ROI-non-relevance, i.e. theabsence of any encoded data relating to the ROI 60, in relation to allpayload packets 32 occurring downstream of the respective ROI packet 64up to the occurrence of the next ROI packet 64 or the end of the currentaccess unit 30 whatever occurs earlier along the sequence 34 of packets.In particular, FIG. 15 illustrates the case where an ROI packet 64 has asyntax element inside, which indicates whether or not the payloadpackets 32 following in the sequential order of packet sequence 34 haveany encoded data relating to the ROI 60 inside or not. Such anembodiment is also described hereinafter. However, another possibilityis, as just mentioned, that each ROI packet 64 indicates merely by itspresence in packet sequence 34 that the payload packet(s) 32 belongingto the “scope” of the respective ROI packet 64, has ROI 60 relating datainside, i.e. data relating to ROI 60. In accordance with an embodimentdescribed hereinafter in more detail, the ROI packet 64 even indicatesthe location of the portion of the ROI 60 encoded into the payloadpacket(s) 32 belonging to its “scope”.

Any network entity 68 receiving the video data stream 22 may exploit theindication of ROI relevance as realized by use of the ROI packets 64 soas to treat, for example, ROI relevant portions of the sequence 34 ofpackets with higher priority than other portions of the packet sequence34, for example. Alternatively, the network entity 68 could use the ROIrelevance information so as to perform other tasks relating to, forexample, the transmission of the video data stream 22. The networkentity 68 may be, for example, a MANE or a decoder for decoding andplaying-back the video content 60 as conveyed via the video data stream22. 28. In other words, network entity 68 may use a result of theidentification of ROI packets so as to decide on transmission taskspertaining the video data stream. The transmission tasks may comprisere-transmission requests concerning defect packets. The network entity68 may be configured to handle the region of interest 70 with increasedpriority and assign a higher priority to ROI packets 72 and theirassociated payload packets, i.e. the ones prefixed by it, which aresignaled as overlaying the region of interest, than compared to ROIpackets and their associated payload packets, which are signaled as notoverlaying the ROI. Network entity 68 may first request a retransmissionof payload packets having the higher priority assigned thereto, beforerequesting any retransmission of payload packets having the lowerpriority assigned thereto.

The embodiment of FIG. 15 may easily be combined with the embodimentdescribed previously with respect to FIGS. 12 to 14. For example, theROI packets 64 mentioned above may also be SEI packets having a certaintype of SEI message contained therein, namely an ROI SEI message. Thatis, an SEI packet may, for example, be a timing control packet andconcurrently an ROI packet, namely if the respective SEI packetcomprises both timing control information as well as ROI indicationinformation. Alternatively, an SEI packet may be one of a timing controlpacket and an ROI packet, rather than the other one, or may be neitheran ROI packet or a timing control packet.

In accordance with the embodiment shown in FIG. 16, interspersing ofpackets between the payload packets of the access units is used toindicate, in a manner easily accessible for network entities 68 handlingthe video data stream 22, which tile or tiles of the current picture 18,which the current access unit 30 relates to, is overlaid by anysub-portion encoded into any of the payload packets 32 for which therespective packets serve as a prefix. In FIG. 16, for example, currentpicture 18 is shown to be sub-divided into four tiles 70, hereexemplarily formed by the four quadrants of the current picture 18. Thesubdivision of the current picture 18 into tiles 70 may, for example, besignaled within the video data stream in units comprising sequences ofpictures such as, for example, in VPS or SPS packets also interspersedinto the sequence 34 of packets. As will be described in more detailbelow, a tile subdivision of current picture 18 may be a regularsubdivision of picture 18 in columns and rows of tiles. The number ofcolumns and the number of rows as well as the width of the columns andthe height of the rows of tiles may be varied. In particular, the widthand height of columns/rows of tiles may be different for different rowsand different columns, respectively. FIG. 16 additionally shows theexample where the sub-portions 24 are slices of picture 18. The slices24 subdivide picture 18. As will be outlined in more detail below,picture's 18 subdivision into slices 24 may be subject to constraintsaccording to which each slice 24 may either be completely containedwithin one single tile 70, or completely cover two or more tiles 70.FIG. 16 illustrates a case where picture 18 is subdivided into fiveslices 24. The first four of these slices 24 in the aforementioneddecoding order cover the first two tiles 70, while the fifth slicecompletely covers the third and fourth tiles 70. Further, FIG. 16illustrates the case where each slice 24 is individually encoded into arespective payload packet 32. Further, FIG. 16 exemplarily illustratesthe case where each payload packet 32 is prefixed by a preceding tileidentification packet 72. Each tile identification packet 72, in turn,indicates for its immediately succeeding payload packet 32 as to whichof the tiles 70 the sub-portion 24 encoded into this payload packet 32overlays. Accordingly, while the first two tile identification packets72 within access unit 30 relating to current picture 18 indicate thefirst tile, the third and fourth tile identification packet 72 indicatethe second tile 70 of picture 18, and the fifth tile identificationpacket 72 indicates the third and fourth tiles 70. With regard to theembodiment of FIG. 16, the same variations are feasible as describedabove with respect to FIG. 15, for example. That is, the “scope” of thetile identification packets 72 may, for example, merely encompass thefirst immediately succeeding payload packet 32 or the immediatelysucceeding payload packets 32 up to the occurrence of the next tileidentification packet.

With regard to the tiles, encoder 10 may be configured to encode eachtile 70 such that, across tile boundaries, no spatial prediction or nocontext selection takes place. Encoder 10 may, for example, encode tile70 in parallel. Likewise, any decoder such as network entity 68 maydecode the tiles 70 in parallel.

The network entity 68 may be a MANE or a decoder or some other device inbetween encoder 10 and decoder, and may be configured to use theinformation conveyed by the tile identification packets 72 to decide oncertain transmission tasks. For example, network entity 68 may handle acertain tile of the current picture 18 of video 16 with higher priority,i.e. may forward the respective payload packets indicated as relating tosuch a tile earlier or using safer FEC protection or the like. In otherwords, the network entity 68 may use a result of the identification soas to decide on transmission tasks pertaining the video data stream. Thetransmission tasks may comprise re-transmission requests concerningpackets received in a defect state—i.e. with exceeding any FECprotection of the video data stream, if any. The network entity mayhandle, for example, different tiles 70 with different priority. To thisend, the network entity may assign a higher priority to tileidentification packets 72 and their payload packets, i.e. the onesprefixed thereby, pertaining to higher priority tiles, than compared totile identification packets 72 and their payload packets pertaining tolower priority tiles. Network entity 68 may, for example, first requesta retransmission of payload packets having the higher priority assignedthereto, before requesting any retransmission of payload packets havingthe lower priority assigned thereto.

The embodiments described so far may be built into the HEVC framework asdescribed in the introductory portion of the specification of thepresent application as described in the following.

In particular, SEI messages may be assigned to slices of decoding unitsin the sub-picture CPB/HRD case. That is, buffering period and timingSEI messages may be assigned to the NAL units containing the slices of adecoding unit. This can be achieved by a new NAL unit type which is anon-VCL NAL unit which is allowed to directly precede one or moreslice/VCL NAL units of a decoding unit. This new NAL unit may be calledslice prefix NAL unit. FIG. 17 illustrates the structure of an accessunit omitting any tentative NAL units for end of sequence and stream.

In accordance with FIG. 17, an access unit 30 is construed as follows:in the sequential order of packets of the sequence 34 of packets, theaccess unit 30 may start with the occurrence of a special type ofpacket, namely an access unit delimiter 80. Then one or more SEI packets82 of an SEI packet type relating to the whole access unit may followwithin the access unit 30. Both packet types 80 and 82 are optional.That is, no packet of this type may occur within an access unit 30.Then, the sequence of decoding units 38 follows. Each decoding unit 38optionally starts with a slice prefix NAL unit 84, including therein forexample timing control information or in accordance with the embodimentof FIG. 15 or 16, an ROI information or tile information or, even moregenerally, a respective sub-picture SEI message 86. Then, the actualslice data 88 in respective payload packets or VCL NAL units follows asindicated in 88. Thus, each decoding unit 38 comprises a sequence of aslice prefix NAL unit 84 followed by respective slice data NAL unit(s)88. The bypass arrow 90 in FIG. 17, bypassing the slice prefix NAL unit,shall indicate that in case of no decoding unit subdivision of thecurrent access unit 30 there may be no slice prefix NAL unit 84.

As already noted above, all information signaled in the slice prefix andassociated sub-picture SEI messages may be either valid for all VCL NALunits in the access unit or until the occurrence of a second prefix NALunit or for the following VCL-NAL unit in decoding order, depending on aflag given in the slice prefix NAL unit.

The slice VCL NAL unit for which the information signaled in the sliceprefix is valid are referred to as prefixed slices in the following.Prefixed slices associated with the a single slice prefixed do notnecessarily constitute a complete decoding unit but can be a part of it.However, a single slice prefix cannot be valid for multiple decodingunits (sub-pictures) and the start of a decoding unit is signaled in theslice prefix. If means for signaling are not given through the sliceprefix syntax (as in the “simple syntax”/version 1 indicated below) theoccurrence of a slice prefix NAL unit signals the start of a decodingunit. Only certain SEI messages (identified via payloadType in thesyntax description below) can be sent exclusively on sub-picture levelwithin the slice prefix NAL unit, while some SEI messages can be senteither in the slice prefix NAL unit on sub-picture level or as a regularSEI message on access unit level.

As discussed above with respect to FIG. 16, additionally oralternatively, a tile ID SEI message/tile ID signaling may be realizedin high level syntax. In earlier designs of a HEVC, the slice header/theslice data contained an identifier for the tile contained in therespective slice. For example, the slice data semantics read:

tile_idx_minus_1 specifies the TileID in raster scan order. The firsttile in the picture shall have a TileID of 0. The value oftile_idx_minus_1 shall be in the range of 0 to(num_tile_columns_minus1+1)*(num_tile_rows_minus1+1)−1.

This parameter however is not considered useful since this ID can beeasily derived from the slice address and the slice dimensions assignaled in the picture parameter set, iftiles_or_entropy_coding_sync_idc is equal to 1.

Although the tile ID can be derived implicitly in the decoding process,the knowledge of this parameter on the application layer is alsoimportant for different use cases such as, for example, in a videoconferencing scenario where different tiles may have different priorityfor the playback (those tiles typically form the region of interestwhich contain the speaker in a conversational use case) may have higherpriority than other tiles. In case of losing network packets in thetransmission of multiple tiles, those network packets containing tilesrepresenting the region of interest may be retransmitted with higherpriority in order to keep the quality of the experience at the receiverterminal higher than in the case retransmitting tiles without anypriority order. Another use case may be to assign tiles, if thedimensions and their position are known, to different screens, e.g. in avideo conferencing scenario.

In order to allow such an application layer to handle tiles with acertain priority in transmission scenarios, the tile_id may be providedas a sub-picture or slice-specific SEI message or in a special NAL unitin front of one or more NAL units of the tile or in a special headersection of the NAL unit belonging to the tile.

As described above with respect to FIG. 15, region of interest SEImessages may also be additionally or alternatively provided. Such an SEImessage could allow the signaling of the region of interest (ROI), inparticular the signaling of an ROI that a certain tile_id/tile belongsto. The message could allow to give region of interest IDs plus apriority of a region of interest.

FIG. 18 illustrates the use of tiles in region of interest signaling.

In addition to what has been described above, slice header signalingcould be implemented. The slice prefix NAL unit may also contain theslice header for the following dependent slices, i.e. the slicesprefixed by the respective slice prefix. If the slice header is onlyprovisioned in the slice prefix NAL unit, the actual slice type needs tobe derived by the NAL unit type of the NAL unit containing therespective dependent slice or by means of a flag in the slice prefixsignaling whether the following slice data belongs to a slice type thatserves as a random access point.

Furthermore, the slice prefix NAL unit may carry slice or sub-picturespecific SEI messages to convey non-mandatory information such assub-picture timing or a tile identifier. Non-mandatory sub-picturespecific messaging is not supported in the HEVC specification describedin the introductory portion of the specification of the presentapplication, but is crucial for certain applications.

In the following, possible syntax for implementing the above-outlinedconcept of slice prefixing is described. In particular, it is describedwhich changes could suffice on a slice level when using the HEVC statusas outlined in the introductory portion of the specification of thepresent application as a basis.

In particular, in the following, two versions of a possible slice prefixsyntax are presented, one with a functionality of SEI messaging only,and one with the extended functionality of signaling a portion of theslice header for the following slices. The first simple syntax/version 1is shown in FIG. 19.

As a preliminary note, FIG. 19 thus shows a possible implementation forimplementing any of the embodiments described above with respect toFIGS. 11 to 16. The interspersed packets shown therein may be construedas shown in FIG. 19 and in the following this is described in moredetail with specific implementation examples.

The extended syntax/version 2 including tile id signaling, decoding unitstart identifier, slice prefix ID and slice header data apart from theSEI message concept is given in the table of FIG. 20.

The semantics could be defined as follows:

rap_flag with a value of 1 indicates that the access unit containing theslice prefix is a RAP picture. rap_flag with a value of 0 indicates thatthe access unit containing the slice prefix is not a RAP picture.

decoding_unit_start_flag indicates the start of a decoding unit withinthe access unit, thus that the following slices up to the end of theaccess unit or the start of another decoding unit belong to the samedecoding unit.

single_slice_flag with a value of 0 indicates that the informationprovided within the prefix slice NAL unit and the associated sub-pictureSEI messages is valid for all following VCL-NAL units until the start ofthe next access unit, the occurrence of another slice prefix or anothercomplete slice header, single_slice_flag with a value 1 indicates thatall information provided in the slice prefix NAL unit and associatedsub-picture SEI messages is valid only for the next VCL-NAL unit indecoding order.

tile_idc indicates the amount of tiles to be present in the followingslice tile_idc equal to 0 indicates that no tiles are used in thefollowing slice. tile_idc equal to 1 indicates that a single tile isused in the following slice and its tile identifier is signaledaccordingly. tile_idc with a value of 2 indicates that multiple tilesare used within the following slice and number of tiles and the firsttile identifier are signaled accordingly.

prefix_slice_header_data_present_flag indicates that data of the sliceheader, corresponding to the slices following in decoding order issignaled in the given slice prefix.

slice_header_data( ) is defined later in the text. It contains therelevant slice header information, which is not covered by the sliceheader, if dependent_slice_flag is set equal to 1.

Note that the decoupling of slice header and actual slice data allowsfor more flexible transmission schemes of header and slice data.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles usedin the following decoding unit minus 1.

first_tile_id_in_prefixed_slices indicates the tile identifier of thefirst tile in the following decoding unit.

For the simple syntax/version 1 of the slice prefix, the followingsyntax elements may be set to default values as follows, if not present:

decoding_unit_start equal to 1, i.e. the slice prefix indicates a startof a decoding unit.

single_slice_flag equal to 0, i.e. the slice prefix is valid for allslices in the decoding unit.

The slice prefix NAL unit is proposed to have a NAL unit type of 24 andthe NAL unit type overview table to be extended according to FIG. 21.

That is, briefly summarizing FIGS. 19 to 21, the syntax details showntherein reveal that a certain packet type may be attributed to the aboveidentified interspersed packets, here exemplarily NAL unit type 24.Moreover, especially the syntax example of FIG. 20 makes it clear thatthe above described alternatives regarding the “scope” of theinterspersed packets, a switching mechanism controlled by a respectivesyntax element within these interspersed packets themselves, hereexemplarily single_slice_flag, may be used in order to control thisscope, i.e. to switch between different alternatives for the definitionof this scope, respectively. Moreover, it has been made clear that theabove described embodiments of FIGS. 1 to 16 may be extended in that theinterspersed packets also comprise common slice header data for slices24 contained in the packets belonging to the “scope” of the respectiveinterspersed packets. That is, there may be a mechanism controlled by arespective flag within these interspersed packets, which indicateswhether common slice header data is contained within the respectiveinterspersed packet or not.

Of course, the concept just presented according to which part of theslice header data is shifted into the slice header prefix, entailschanges to the slice headers as specified in the HEVC's current version.The table in FIG. 22 shows a possible syntax for such a slice header,where certain syntax elements present in the slice header according tothe current version, are shifted to a lower hierarchy syntax element,referred to as slice_header_data( ). This syntax of the slice header andthe slice header data only applies to the option according to which theextended slice header prefix NAL unit concept is used.

In FIG. 22, slice_header_data_present_flag indicates that slice headerdata for the present slice shall be predicted from the values signaledin the last slice prefix NAL unit in the access unit, i.e. the mostrecently occurring slice prefix NAL unit.

All syntax elements removed from the slice header are signaled throughthe syntax element slice header data as given in the table of FIGS. 23Ato 23C.

That is, transferring the concept of FIG. 22 and FIGS. 23A to 23C ontothe embodiments of FIGS. 12 to 16, the interspersed packets describedtherein may be extended by the concept of incorporating into theseinterspersed packets, a part of the slice header syntax of the slices(sub-portions) 24 encoded into the payload packets, i.e. VCL NAL units.The incorporation may be optional. That is, a respective syntax elementin the interspersed packet may indicate whether such slice header syntaxis contained in the respective interspersed packet or not. Ifincorporated, the respective slice header data incorporated into arespective interspersed packet may apply to all slices contained in thepacket belonging to the “scope” of the respective interspersed packet.Whether the slice header data contained in an interspersed packet isadopted by a slice encoded into any of the payload packets belonging tothe scope of this interspersed packet, may be signaled by a respectiveflag, such as slice_header_data_present_flag of FIG. 22. By thismeasure, the slice headers of the slices encoded into the packetsbelonging to the “scope” of a respective interspersed packet may bedownsized accordingly using the just mentioned flag in the slices sliceheader and any decoder receiving the video data stream, such as thenetwork entities shown in the above FIGS. 12 to 16, would be responsiveto the just mentioned flag in the slices slice header so as to copy theslice header data incorporated into an interspersed packet into theslice header of a slice encoded into a payload packet belonging to thescope of this interspersed packet in case of the respective flag withinthat slice signaling the displacement of the slice header data to theslice prefix, i.e. the respective interspersed packet.

Proceeding further with the syntax example for implementing theembodiments of FIGS. 12 to 16, the SEI message syntax may be as shown inFIG. 24. In order to introduce slice or sub-picture SEI message types,the SEI payload syntax may be adapted as presented in the table of FIGS.25A and 25B. Only the SEI message with payloadType in the range of 180to 184 may be sent exclusively on sub-picture level within the sliceprefix NAL unit. Additionally, the region of interest SEI messages withpayloadType equal to 140 can be sent either in the slice prefix NAL uniton sub-picture level, or the regular SEI message on access unit level.

That is, in transferring the details shown in FIGS. 25 and 24 onto theembodiments described above with respect to FIGS. 12 to 16, theinterspersed packets shown in these embodiments of FIGS. 12 to 16 may berealized by using slice prefix NAL units with a certain NAL unit type,e.g. 24, comprising a certain type of SEI message signaled bypayloadType at the beginning of each SEI message within the slice prefixNAL unit, for example. In the specific syntax embodiment described now,payloadType=180 and payloadType=181 results in a timing control packetin accordance with the embodiments of FIGS. 11 to 14, whilepayloadType=140 results in an ROI packet in accordance with theembodiment of FIG. 15, and payloadType=182 results in a tileidentification packet in accordance with the embodiment of FIG. 16. Thespecific syntax example described herein below may comprise merely oneor a subset of the just mentioned payloadType options. Beyond this,FIGS. 25A and 25B reveal that any of the above described embodiments ofFIGS. 11 to 16 may be combined with each other. Even further, FIGS. 25Aand 25B reveal that any of the above embodiments of FIGS. 12 to 16, orany combination thereof, may be extended by a further interspersedpacket, subsequently explained with payloadType=184. As alreadydescribed above, an extension described below with respect topayloadType=183 ends-up in the possibility that any interspersed packetsmay have incorporated thereinto common slice header data for sliceheaders of slices encoded into any payload packet belonging to itsscope.

The tables in the following figures define SEI messages which may beused on slice or sub-picture level. A region of interest SEI message isalso presented which may be used on sub-picture and access unit level.

FIG. 26, for example, shows an example for a sub-picture buffering SEImessage occurring whenever a slice prefix NAL unit of NAL unit type 24has an SEI message type 180 contained therein, thus forming a timingcontrol packet.

The semantics could be defined as follows:

seq_parameter_set_id specifies the sequence parameter set that containsthe sequence HRD attributes. The value of seq_parameter_set_id shall beequal to the value of seq_parameter_set_id in the picture parameter setreferenced by the primary coded picture associated with the bufferingperiod SEI message. The value of seq_parameter_set_id shall be in therange of 0 to 31, inclusive.

initial_cpb_removal_delay[SchedSelIdx] andinitial_alt_cpb_removal_delay[SchedSelIdx] specify the initial CPBremoval delays for the SchedSelIdx-th CPB of the decoding unit (thesub-picture). The syntax elements have a length in bits given byinitial_cpb_removal_delay_length_minus1+1, and are in units of a 90 kHzclock. The values of the syntax elements shall not be equal to 0 andshall not exceed 90000*(CpbSize[SchedSelIdx]÷BitRate[SchedSelIdx]), thetime-equivalent of the CPB size in 90 kHz clock units.

Over the entire coded video sequence, the sum ofinitial_cpb_removal_delay[SchedSelIdx] andinitial_cpb_removal_delay_offset[SchedSelIdx] per decoding unit(sub-picture) shall be constant for each value of SchedSelIdx, and thesum of initial_alt_cpb_removal_delay[SchedSelIdx] andinitial_alt_cpb_removal_delay_offset[SchedSelIdx] shall be constant foreach value of SchedSelIdx.

FIG. 27 shows likewise an example for a sub-picture timing SEI message,wherein the semantics could be described as follows:

du_cpb_removal_delay specifies how many clock ticks to wait afterremoval from the CPB of the decoding unit (sub-picture) associated withthe most recent sub-picture buffering period SEI message in a precedingaccess unit in the same decoding unit (sub-picture), if present,otherwise associated with the most recent buffering period SEI messagein a preceding access unit, before removing from the buffer the decodingunit (sub-picture) data associated with the sub-picture timing SEImessage. This value is also used to calculate an earliest possible timeof arrival of decoding unit (sub-picture) data into the CPB for the HSS(Hypothetical Stream Scheduler [2]0). The syntax element is a fixedlength code whose length in bits is given bycpb_removal_delay_length_minus1+1. The cpb_removal_delay is theremainder of a modulo 2^((cpb_removal_delay_length_minus1+1)) counter.

du_dpb_output_delay is used to compute the DPB output time of thedecoding unit (sub-picture). It specifies how many clock ticks to waitafter removal of the decoded decoding unit (sub-picture) from the CPBbefore the decoding unit (sub-picture) of picture is output from theDPB.

Note that this allows for sub-picture updates. In such a scenario, thenon-updated decoding units may remain unchanged of the last decodedpicture, i.e. they remain visible.

Summarizing FIGS. 26 and 27 and transferring the specific detailscontained therein onto the embodiment of FIGS. 12 to 14, it may be saidthat the decoder buffer retrieval time for a decoding unit may besignaled in the associated timing control packet in a differentiallycoded manner, namely incrementally relative to another decoding bufferretrieval time. That is, in order to obtain the decoder buffer retrievaltime for a certain decoding unit, a decoder receiving the video datastream adds the decoder retrieval time obtained from the timing controlpacket prefixing the certain decoding unit, to the decoder retrievaltime of the immediately preceding decoding unit, i.e. the one precedingthe certain decoding unit, and proceeding in this manner with thefollowing decoding units. At the beginnings of coded video sequences ofseveral pictures each or parts thereof, a timing control packet mayadditionally or alternatively comprise a decoder buffer retrieval timevalue coded absolutely rather than differentially relative to anypreceding decoding unit's decoder buffer retrieval time.

FIG. 28 shows how the sub-picture slice info SEI message may look like.The semantics could be defined as follows:

slice_header_data _flag with a value of 1 indicates that slice headerdata is present in the SEI message. The slice header data provided inthe SEI is valid for all slices following in decoding order until theend of the access unit, the occurrence of slice data in another SEImessage, slice NAL unit or slice prefix NAL unit.

FIG. 29 shows an example for a sub-picture tile info SEI message,wherein the semantics could be defined as follows:

tile_priority indicates the priority of all tiles in the prefixed slicesfollowing in decoding order. The value of tile_priority shall be in therange of 0 to 7 inclusively, where 7 indicates the highest priority.

multiple_tiles_in_prefixed_slices_flag with a value of 1 indicates thatthere are more than 1 tiles in the prefixed slices to follow in decodingorder. multiple_tiles_in_prefixed_slices_flag with a value of 0indicates that the following prefixed slices contain only one tile.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles in theprefixed slices following in decoding order.

first_tile_id_in_prefixed_slices indicates the tile_id of the first tilein the prefixed slices following in decoding order.

That is, the embodiment of FIG. 16 could be implemented using the syntaxof FIG. 29 for realizing the tile identification packets mentioned inFIG. 16. As shown therein, a certain flag, heremultiple_tiles_in_prefixed_slices_flag, may be used to signal within theinterspersed tile identification packet whether merely one tile or morethan one tile is covered by any sub-portion of the current picture 18encoded into any of the payload packets belonging to the scope of therespective interspersed tile identification packet. If the flag signalsthe overlaying of more than one tile, a further syntax element iscontained in the respective interspersed packet, here exemplarilynum_tiles_in_prefixed_slices_minus1 indicating the number of tilesoverlaid by any sub-portion of any payload packet belonging to the scopeof the respective interspersed tile identification packet. Finally, afurther syntax element, here exemplarilyfirst_tile_id_in_prefixed_slices, indicates the ID of the tile among thenumber of tiles indicated by the current interspersed tileidentification packet, which is the first in accordance with thedecoding order. Transferring the syntax of FIG. 29 onto the embodimentof FIG. 16, the tile identification packet 72 prefixing the fifthpayload packet 32 could, for example, have all three just-discussedsyntax elements with multiple_tiles_in_prefixed_slices_flag being set to1, num_tiles_in_prefixed_slices_minus1 being set to 1, therebyindicating that two tiles belong to the current scope, andfirst_tile_id_in_prefixed_slices being set to 3, indicating that the runof tiles in decoding order belonging to the scope of the current tileidentification packet 72 starts at the third tile (having tile_id=2).

FIG. 29 also reveals that a tile identification packet 72 may possiblyalso indicate a tile_priority, i.e. a priority of the tiles belonging toits scope. Similar to the ROI aspect, the network entity 68 may use suchpriority information to control transmission tasks such as the requestfor retransmissions of certain payload packets.

FIG. 30 shows a syntax example for a sub-picture tile dimension info SEImessage, wherein the semantics could be defined as:

multiple_tiles_in_prefixed_slices_flag with a value of 1 indicates thatthere are more than 1 tiles in the prefixed slices to follow in decodingorder. multiple_tiles_in_prefixed_slices_flag with a value of 0indicates that the following prefixed slices contain only one tile.

num_tiles_in_prefixed_slices_minus1 indicates the number of tiles in theprefixed slices following in decoding order.

tile_horz_start[i] indicates the start in horizontal direction of thei-th tile in pixels within the picture.

tile_width[i] indicates the width of the i-th tile in pixels within thepicture.

tile_vert_start[i] indicates the start in horizontal direction of thei-th tile in pixels within the picture.

tile_height[i] indicates the height of the i-th tile in pixels withinthe picture.

Note that the tile dimension SEI message may be used to in displayoperations, e.g., for assigning a tile to a screen in multiple screendisplay scenario.

FIG. 30 thus reveals that the implementation syntax example of FIG. 29with regard to the tile identification packets of FIG. 16 may be variedin that the tiles belonging to the scope of the respective tileidentification packet are indicated by their location within the currentpicture 18 rather than their tile ID. That is, rather than signaling thetile ID of the first tile in decoding order covered by the respectivesub-portions encoded into any of the payload packet belonging to thescope of the respective interspersed tile identification packet, foreach tile belonging to the current tile identification packet, itsposition could be signaled by signaling, for example, the upper leftcorner position of each tile i, here exemplarily by tile_horz_start andtile_vert_start, and the width and height of tile i, here exemplarily bytile_width and tile_height.

A syntax example for a region of interest SEI message is shown in FIG.31. To be even more precise, FIG. 32 shows a first variant. Inparticular, the region of interest SEI message may, for example, be usedon access unit level or on sub-picture level to signal one or moreregions of interest. In accordance with the first variant of FIG. 32, anindividual ROI is signaled once per ROI SEI message, rather thansignaling all ROIs of the respective ROI packet's scope within one ROISEI message if multiple ROIs are within the current scope.

In accordance with FIG. 31, the region of interest SEI message signalseach ROI individually. The semantics could be defined as follows:

roi_id indicates the identifier of the region of interest.

roi_priority indicates the priority of all tiles that belongs to theregion of interest in the prefixed slices or all slices following indecoding order depending on whether the SEI message is sent onsub-picture level or access unit level. The value of roi_priority shallbe in the range of 0 to 7 inclusively, where 7 indicates the highestpriority. If both, roi_priority in the roi info SEI message andtile_priority in the sub-picture tile info SEI messages are given, thehighest value of both is valid for the priority of the individual tiles.

num_tiles_in_roi_minus1 indicates the number of tiles in the prefixedslices following in decoding order that belong to the region ofinterest.

roi_tile_id[i] indicates the tile_id of the i-th tile that belongs tothe region of interest in the prefixed slices following in decodingorder.

That is, FIG. 31 shows that an ROI packet as shown in FIG. 15 couldsignal therein an ID of the region of interest which the respective ROIpacket and the payload packet belonging to its scope refer to.Optionally, an ROI priority index may be signaled along with the ROI ID.However, both syntax elements are optional. Then, a syntax elementnum_tiles_in_roi_minus1 may indicate the number of tiles in the scope ofthe respective ROI packet belonging to the respective ROI 60. Then,roi_tile_id indicates the tile-ID of the i-th tiles belonging to the ROI60. Imagine, for example, picture 18 would be subdivided into tiles 70in the way shown in FIG. 16, and that the ROI 60 of FIG. 15 wouldcorrespond to the left-hand half of picture 18, would be formed by thefirst and third tile in decoding order. Then, a ROI packet may be placedin front of the first payload packet 32 of access unit 30 in FIG. 16,followed by a further ROI packet between the fourth and fifth payloadpackets 32 of this access unit 30. Then, the first ROI packet would havenum_tile_in_roi_minus1 be set to 0 with roi_tile_id[0] being 0 (therebyreferring to the first tile in decoding order), wherein the second ROIpacket in front of the fifth payload packet 32 would havenum_tiles_in_roi_minus1 being set to 0 with roi_tile_id[0] being set to2 (thereby denoting the third tile in decoding order at the left-handbottom quarter of picture 18).

According to the second variant, the syntax of a region of interest SEImessage could be as shown in FIG. 32. Here, all ROIs in a single SEImessage are signaled. In particular, the same syntax as discussed abovewith respect to FIG. 31 would be used, but multiplying the syntaxelements for each of ROIs of a number of ROIs which the respective ROISEI message or ROI packet refers to, with a number being signaled by asyntax element, here exemplarily num_rois_minus1. Optionally, a furthersyntax element, here exemplarily roi_presentation_on_separate_screen,could signal for each ROI whether the respective ROI is suitable forbeing presented on a separate screen.

The semantic could be as follows:

num_rois_minus1 indicates the number of ROIs in the prefixed slices orregular slices following in decoding order.

roi_id[i] indicates the identifier of the i-th region of interest.

roi_priority[i] indicates the priority of all tiles that belongs to thei-th region of interest in the prefixed slices or all slices followingin decoding order depending on whether the SEI message is sent onsub-picture level or access unit level. The value of roi_priority shallbe in the range of 0 to 7 inclusively, where 7 indicates the highestpriority. If both, roi_priority in the roi_info SEI message andtile_priority in the sub-picture tile info SEI messages are given, thehighest value of both is valid for the priority of the individual tiles.

num_tiles_in_roi_minus1[i] indicates the number of tiles in the prefixedslices following in decoding order that belong to the i-th region ofinterest.

roi_tile_id[i][n] indicates the tile_id of the n-th tile that belongs tothe i-th region of interest in the prefixed slices following in decodingorder.

roi_presentation_on_seperate_screen [i] indicates that the region ofinterest, associated with the i-th roi_id is suitable for presentationon a separate screen.

Thus, briefly summarizing the various embodiments described so far, anadditional high level syntax signaling strategy has been presented whichallows to apply SEI messages as well as additional high level syntaxitems beyond the ones included in the NAL unit header on a per slicelevel. Therefore, we described the slice prefix NAL unit. The syntax andsemantics of the slice prefix and slice_level/sub-picture SEI messageshas been described along with use cases for low delay/sub-picture CPBoperations, tile signaling and ROI signaling. An extended syntax hasbeen presented to signal part of the slice header of following slices inthe slice prefix additionally.

For the sake of completeness, FIG. 33 shows a further example for asyntax which could be used for a timing control packet according to theembodiment of FIGS. 12 to 14. The semantics could be:

du_spt_cpb_removal_delay_increment specifies the duration, in units ofclock sub-ticks, between the nominal CPB times of the last decoding unitin decoding order in the current access unit and the decoding unitassociated with the decoding unit information SEI message. This value isalso used to calculate an earliest possible time of arrival of decodingunit data into the CPB for the HSS, as specified in Annex C. The syntaxelement is represented by a fixed length code whose length in bits isgiven by du_cpb_removal_delay_increment_length_minus1+1. When thedecoding unit associated with the decoding unit information SEI messageis the last decoding unit in the current access unit, the value ofdu_spt_cpb_removal_delay_increment shall be equal to 0.

dpb_output_du_delay_present_flag equal to 1 specifies the presence ofthe pic_spt_dpb_output_du_delay syntax element in the decoding unitinformation SEI message. dpb_output_du_delay_present_flag equal to 0specifies the absence of the pic_spt_dpb_output_du_delay syntax elementin the decoding unit information SEI message.

pic_spt_dpb_output_du_delay is used to compute the DPB output time ofthe picture when SubPicHrdFlag is equal to 1. It specifies how many subclock ticks to wait after removal of the last decoding unit in an accessunit from the CPB before the decoded picture is output from the DPB.When not present, the value of pic_spt_dpb_output_du_delay is inferredto be equal to pic_dpb_output_du_delay. The length of the syntax elementpic_spt_dpb_output_du_delay is given in bits bydpb_output_delay_du_length_minus1+1.

It is a requirement of bitstream conformance that all decoding unitinformation SEI messages that are associated with the same access unit,apply to the same operation point, and havedpb_output_du_delay_present_flag equal to 1 shall have the same value ofpic_spt_dpb_output_du_delay. The output time derived from thepic_spt_dpb_output_du_delay of any picture that is output from an outputtiming conforming decoder shall precede the output time derived from thepic_spt_dpb_output_du_delay of all pictures in any subsequent CVS indecoding order.

The picture output order established by the values of this syntaxelement shall be the same order as established by the values ofPicOrderCntVal.

For pictures that are not output by the “bumping” process because theyprecede, in decoding order, an IRAP picture with NoRaslOutputFlag equalto 1 that has no_output_of_prior_pics_flag equal to 1 or inferred to beequal to 1, the output times derived from pic_spt_dpb_output_du_delayshall be increasing with increasing value of PicOrderCntVal relative toall pictures within the same CVS. For any two pictures in the CVS, thedifference between the output times of the two pictures whenSubPicHrdFlag is equal to 1 shall be identical to the same differencewhen SubPicHrdFlag is equal to 0.

Further, FIG. 34 shows a further example for signaling an ROI regionusing ROI packets. In accordance with FIG. 34, the syntax of an ROIpacket comprises merely one flag indicating whether all sub-portions ofpicture 18 encoded into any payload packet 32 belonging to its scopebelongs to the ROI or not. The “scope” extends up to the occurrence ofthe ROI packet or region_refresh_info SEI message. If the flag is 1, theregion is indicated as being encoded into a respective subsequentpayload packet(s), and if 0 the opposite applies, i.e. the respectivesub-portions of picture 18 do not belong to the ROI 60.

Before discussing some of the above embodiments again in other wordswith additionally explaining some terms used above such as tile, sliceand WPP sub-stream sub-divisioning, it should be noted that the aboveembodiments High Level signaling may alternatively be defined intransport specifications such as [3-7]. In other words, the packetsmentioned above and forming sequence 34 may be transport packets some ofwhich having the application layer's sub-portions such as slices,incorporated, such as packetized in full or fragmented, thereinto, somebeing interspersed between the latter in the manner, and with the aim,discussed above. In other words, above-mentioned interspersed packetsare not restricted to be defined as SEI massages of other types of NALunits, defined in the application layer's video codec, but couldalternatively be extra transport packet defined in transport protocols.

In other words, in accordance with one aspect of the presentspecification, above embodiments revealed a video data stream havingvideo content encoded therein in units of sub-portions (see coding treeblocks or slices) of pictures of the video content, each sub-portionbeing respectively encoded into one or more payload packets (see VCL NALunits) of a sequence of packets (NAL units) of the video data stream,the sequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets relating to arespective picture of the video content, wherein the sequence of packetshas interspersed thereinto timing control packets (slice prefix) so thatthe timing control packets subdivide the access units into decodingunits so that at least some access units are subdivided into two or moredecoding units, with each timing control packet signaling a decoderbuffer retrieval time for a decoding unit, the payload packets of whichfollow the respective timing control packet in the sequence of packets.

As described above, the domain with respect to which the video contentis encoded into the data stream in units of sub-portions of pictures,may cover the syntax elements relating to predictive coding such ascoding modes (such as intra mode, inter mode, sub-division informationand the like), prediction parameters (such as motion vectors,extrapolation directions or the like) and/or residual data (such astransform coefficient levels, with these syntax elements beingassociated with local portions of the picture such as coding treeblocks, prediction blocks and residual (such as transform) blocks,respectively.

As described above, the payload packets may each encompass one or moreslices (in complete, respectively). The slices may be independentlydecodable or may show interrelations which hinder an independentdecoding thereof. For example, entropy slices may be independentlyentropy decodable, but prediction beyond the slice boundaries may beprohibited. Dependent slices may allow for WPP processing, i.e.coding/decoding using entropy and predictive coding beyond the sliceboundaries with the ability of parallely coding/decoding the dependentslices in a time overlapping manner with, however, a staggered commenceof the coding/decoding procedure of the individual dependent slices andthe slice/slices referred to by the dependent slices.

The sequential order at which the payload packets of an access unit arearranged within the respective access unit may be known to the decoderin advance. For example, a coding/decoding order may be defined amongthe sub-portions of the pictures such as the scanning order among thecoding tree blocks in the above examples.

See, for example, the figure below. A currently coded/decoded picture100 may be divided into tiles which, in FIGS. 35 and 36, for example,exemplarily correspond to the four quadrants of the picture 110 and areindicated with reference signs 112 a-112 d. That is, the whole picture110 may form one tile as in case of FIG. 37 or may be segmented intomore than one tile. The tile segmentation may be restricted to regularones in which the tiles are arranged in columns and rows only. Differentexamples are presented below.

As can be seen, the picture 110 is further subdivided into coding (tree)blocks (small boxes in the figure and called CTB above) 114 among whicha coding order 116 is defined (here, raster scan order, but may also bedifferent). The picture's sub-division into the tiles 112 a-d may berestricted so that tiles are disjoint sets of the blocks 114.Additionally, both blocks 114 and tiles 112 a-d may be restricted to aregular arrangement in columns and rows.

If tiles (i.e. more than one) are present, then the (de)coding order 116raster scans a first complete tile first with then transitioning—also ina raster scan tile order—to the next tile in tile order.

As tiles are en/decodable independent from each other due to thenon-crossing of tile boundaries by spatial predictions and contextselections deduced from spatial neighborhood, encoder 10 and decoder 12may encode/decode a picture sub-divided into tiles 112 (formerlyindicated by 70), in parallel, independent from each other—except for,for example, an in-loop or post-filtering which may be allowed to crosstile boundaries.

The picture 110 may further be subdivided into slices 118 a-d,180—formerly indicated using reference sign 24. A slice may containmerely a part of a tile, one complete tile or more than one tiles incomplete. Thus, the division into slices may also subdivide tiles as incase of FIG. 35. Each slice comprises at least one coding block 114 incomplete and is composed of consecutive coding blocks 114 in codingorder 116 so that an order is defined among the slices 118 a-d followingwhich the indices in the figure have been assigned. The slice divisionin FIGS. 35 to 37 has been chosen for illustration purposes only. Thetile boundaries may signaled in the data stream. The picture 110 mayform a single tile as depicted in FIG. 37.

Encoder 10 and decoder 12 may be configured to obey tile boundaries inthat spatial prediction is not applied across tile boundaries. Thecontext adaptation, i.e. probability adaptations of the various entropy(arithmetic) contexts may continue over whole slices. However, whenevera slice crosses—along coding order 116—a tile boundary (if presentwithin the inner of a slice) such as in FIG. 36 with regard to slices118 a,b, then the slice is, in turn, subdivided into subsections(substreams or tiles) with the slice comprising pointers (c.p. entrypoint offset) pointing to the beginnings of each subsection. Indecoder-loop filters may be allowed to cross tile boundaries. Suchfilters may involve one or more of a deblocking filter, a SampleAdaptive Offset (SAO) filter and an Adaptive loop filter (ALF). Thelatter may be applied over tile/slice boundaries if activated.

Each optional second and following subsections may have their beginningpositioned byte-aligned within the slice with the pointer indicating theoffset from beginning of one subsection to the beginning to the nextsubsection. The subsections are arranged within slices in the scan order116. FIG. 38 shows an example with slice 180 c of FIG. 37 beingexemplarily subdivided into subsections 119 _(i).

With regard to the figures it is noted that slices forming subparts oftiles do not have to end with the row in the tile 112 a. See, forexample slice 118 a in FIGS. 37 and 38.

The figure below shows an exemplary portion of a data stream relating toan access unit associated with the picture 110 of the above FIG. 38).Here, each payload packet 122 a-d—formerly indicated by reference sign32—exemplarily accommodates merely one slice 118 a. Two timing controlpackets 124 a,b—formerly indicated by reference sign 36—are shown asbeing interspersed into the access unit 120 for illustration purposes:124 a precedes packet 122 a in packet order 126 (corresponding tode/encoding time axis) and 124 b precedes packet 122 c. Accordingly,access unit 120 is divided into two decoding units 128 a,b formerlyindicated by reference sign 38—, the first one of which comprisespackets 122 a,b (along with optional filler data packets (succeeding thefirst and second packets 122 a,b, respectively) and optional access unitleading SEI packets (preceding the first packet 122 a)) and the secondone of which comprises packets 118 c,d (along with optional filler datapackets (succeeding packets 122 c,d, respectively)).

As described above, each packet of the sequence of packets may beassigned to exactly one packet type out of a plurality of packet types(nal_unit_type). Payload packets and timing control packets (andoptional filler data and SEI packets) are, for example, of differentpacket types. The instantiations of packets of a certain packet type inthe sequence of packets may be subject to certain limitations. Theselimitations may define an order among the packet types (see FIG. 17)which is to be obeyed by the packets within each access unit so thataccess unit borders 130 a,b are detectable, and remain at the sameposition within the sequence of packets, even if packets of anyremovable packet type are removed from the video data stream. Forexample, payload packets are of the non-removable packet type. However,timing control packets, filler data packets and SEI packets may, asdiscussed above, be of the removable packet type, i.e. they may benon-VCL NAL units.

In the above example, timing control packets have explicitly beenexemplified above by the syntax of slice_prefix_rbsp( ).

Using such an interspersing of timing control packets, an encoder isenabled to adjust the buffering scheduling at the decoder side duringthe course of encoding the individual pictures of the video content. Forexample, the encoder is enabled to optimize the buffer scheduling tominimize the end-to-end delay. In this regard, the encoder is enabled totake the individual distribution of coding complexness across thepicture area of the video content for the individual pictures of thevideo content into account. In particular, the encoder may continuouslyoutput the sequence of packets 122, 122 a-d, 122 a-d ₁₋₃ on apacket-by-packet basis (i.e. as soon as a current packet has beenfinalized it is output). By use of the timing control packets, theencoder is able to adjust the buffer scheduling at the decoding side atmoments where some of the sub-portions of the current picture havealready been encoded into respective payload packets with remainingsub-portions, however, not yet having been encoded.

Accordingly, an encoder for encoding into a video data stream videocontent in units of sub-portions (see coding tree blocks, tiles orslices) of pictures of the video content, with respectively encodingeach sub-portion into one or more payload packets (see VCL NAL units) ofa sequence of packets (NAL units) of the video data stream so that thesequence of packets is divided into a sequence of access units and eachaccess unit collects the payload packets relating to a respectivepicture of the video content, may be configured to intersperse into thesequence of packets timing control packets (slice prefix) so that thetiming control packets subdivide the access units into decoding units sothat at least some access units are subdivided into two or more decodingunits, with each timing control packet signaling a decoder bufferretrieval time for a decoding unit, the payload packets of which followthe respective timing control packet in the sequence of packets.

Any decoder receiving the just-outlined video data stream is free toexploit the scheduling information contained in the timing controlpacket or not. However, while the decoder is free to exploit theinformation, a decoder conforming with the codec level is able to decodedata following the indicated timing. If exploitation takes place, thedecoder feeds its decoder buffer and empties its decoder buffer in unitsof decoding units. The “decoder buffer” may, as described above, involvethe decoded picture buffer and/or the coded picture buffer.

Accordingly, a decoder for decoding a video data stream having videocontent encoded therein in units of sub-portions (see coding treeblocks, tiles or slices) of pictures of the video content, eachsub-portion being respectively encoded into one or more payload packets(see VCL NAL units) of a sequence of packets (NAL units) of the videodata stream, the sequence of packets being divided into a sequence ofaccess units so that each access unit collects the payload packetsrelating to a respective picture of the video content, may be configuredto look for timing control packets interspersed into the sequence ofpackets, subdivide the access units into decoding units at the timingcontrol packets so that at least some access units are subdivided intotwo or more decoding units, derive from each timing control packet adecoder buffer retrieval time for a decoding unit, the payload packetsof which follow the respective timing control packet in the sequence ofpackets, and retrieve the decoding units from a buffer of the decoderscheduled at times defined by the decoder buffer retrieval times for thedecoding units.

Looking for the timing control packet may involve the decoder inspectingthe NAL unit header and the syntax element comprised thereby, namelynal_unit_type. If the value of the latter flag equals some value, i.e.is, in accordance with the above examples, 124, then the packetcurrently inspected is a timing control packet. That is, the timingcontrol packet would comprise or convey the information explained abovewith respect to pseudo code subpic_buffering as well as subpic_timing.That is, the timing control packets may convey or specify initial CPBremoval delays for the decoder or specify how many clock ticks to waitafter removal from the CPB of a respective decoding unit.

In order to allow for a repetitive transmission of the timing controlpackets without unintentionally further dividing the access unit intofurther decoding units, a flag within the timing control packets mayexplicitly signal whether the current timing control packet participatesin the access unit subdivision into coding units or not (comparedecoding_unit_start_flag=1 indicating the start of a decoding unit, anddecoding_unit_start_flag=0 signaling the opposite circumstance).

The aspect of using interspersed decoding unit related tileidentification information differs from the aspect of using intersperseddecoding unit related timing control packets in that tile identificationpackets are interspersed into the data stream. The above-mentionedtiming control packets may additionally be interspersed into the datastream or the decoder buffer retrieval times may be conveyed along withthe below explained tile identification information within the samepacket commonly. Accordingly, details brought forward in the abovesection may be used in order to clarify issues in the description below.

A further aspect of the present specification derivable from theabove-described embodiments reveals a video data stream having videocontent encoded therein, using predictive and entropy coding, in unitsof slices into which pictures of the video content are spatiallysubdivided, using a coding order among the slices, with restrictingpredictions of the predictive coding and/or entropy coding to the innerof tiles into which the pictures of the video content are spatiallysubdivided, wherein the sequence of the slices in coding order arepacketized into payload packets of a sequence of packets (NAL units) ofthe video data stream in the coding order, the sequence of packets beingdivided into a sequence of access units so that each access unitcollects the payload packets having packetized thereinto slices relatingto a respective picture of the video content, wherein the sequence ofpackets has tile identification packets interspersed thereintoidentifying tiles (potentially merely one) which are overlaid by slices(potentially merely one) packetized into one or more payload packetsimmediately following the respective tile identification packet in thesequence of packets.

See, for example, the immediately preceding figure showing a datastream. Packets 124 a and 124 b shall now represent tile identificationpackets. Either by explicit signaling (compare single_slice_flag=1) orper convention, the tile identification packet may merely identify tileswhich are overlaid by slices packetized into the immediately followingpayload packet 122 a. Alternatively, by explicit signaling or perconvention, the tile identification packet 124 a may identify tileswhich are overlaid by slices packetized into one or more payload packetsimmediately following the respective tile identification packet 124 a inthe sequence of packets until the earlier of the end 130 b of thecurrent access unit 120, and the starting of a next decoding unit 128 b,respectively. See, for example, FIG. 35: if each slice 118 a-d ₁₋₃ wasseparately packetized into a respective packet 122 a-d ₁₋₃, with thesubdivision into decoding units being such that the packets are groupedinto three decoding units according to {122 a ₁₋₃}, {122 b ₁₋₃} and {122c ₁₋₃, 122 d ₁₋₃}, then the slices {118 c ₁₋₃, 118 d ₁₋₃} packetizedinto the packets {122 c ₁₋₃, 122 d ₁₋₃} of the third decoding unitwould, for example, overlay tiles 112 c and 112 d, and the correspondingslice prefix would, for example, when referring to the complete decodingunit, indicate “c” and “d”, i.e. these tiles 112 c and 112 d.

Thus, the network entity mentioned further below may use this explicitsignaling or convention in order to correctly associate each tileidentification packet with one or more payload packets immediatelyfollowing the identification packet in the sequence of packets. The waythe identification may be signaled has exemplarily been described aboveby way of the pseudo code subpic_tile_info. The associated payloadpackets were mentioned above as “prefixed slices”. Naturally, theexample may be modified. For example, the syntax elements“tile_priority” may be left away. Further, the order among the syntaxelements may be switched and the descriptor regarding possible bitlengths and encoding principles of the syntax elements is merelyillustrative.

A network entity which receives the video data stream (i.e. a video datastream having video content encoded therein, using predictive andentropy coding, in units of slices into which pictures of the videocontent are spatially subdivided, using a coding order among the slices,with restricting predictions of the predictive coding and/or entropycoding to the inner of tiles into which the pictures of the videocontent are spatially subdivided, wherein the sequence of the slices incoding order are packetized into payload packets of a sequence ofpackets (NAL units) of the video data stream in the coding order, thesequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets having packetizedthereinto slices relating to a respective picture of the video content,wherein the sequence of packets has tile identification packetsinterspersed thereinto) may be configured to identify, based on the tileidentification packets, tiles which are overlaid by slices packetizedinto one or more payload packets immediately following the respectivetile identification packet in the sequence of packets. The networkentity may use the identification result so as to decide on transmissiontasks. For example, the network entity may handle the different tileswith different priority for playback. For example, in case of packetloss those payload packets relating to tiles of higher priority may beadvantageous for a retransmission over payload packets relating to tilesof lower priority. That is, the network entity may first request theretransmission of lost payload packets relating to tiles of higherpriority. Merely in case of enough time being left (depending on thetransmission rate) the network entity proceeds with requesting theretransmission of lost payload packets relating to tiles of lowerpriority. The network entity may, however, also be a playback unit whichis able to assign tiles or payload packets relating to certain tiles todifferent screens.

With regard to the aspect of using interspersed region of interestinformation, it should be noted that the ROI packets mentioned belowcould coexist with the above mentioned timing control packets and/ortile identification packets, either by combining the information contentthereof within common packets as described above with respect to theslice prefixes, or in the form of separate packets.

The aspect of using interspersed region of interest information asdescribed above reveals, in other words, a video data stream havingvideo content encoded therein, using predictive and entropy coding, inunits of slices into which pictures of the video content are spatiallysubdivided, using a coding order among the slices, with restrictingpredictions and/or entropy coding of the predictive coding to the innerof tiles into which the pictures of the video content are divided,wherein the sequence of the slices in coding order are packetized intopayload packets of a sequence of packets (NAL units) of the video datastream in the coding order, the sequence of packets being divided into asequence of access units so that each access unit collects the payloadpackets having packetized thereinto slices relating to a respectivepicture of the video content, wherein the sequence of packets has ROIpackets interspersed thereinto identifying tiles of the pictures whichbelong to a ROI of the pictures, respectively.

With regard to the ROI packets, similar comments are valid as thoseprovided before with respect to the tile identification packets: the ROIpackets may identify tiles of the pictures which belong to an ROI of thepicture merely among those tiles which are overlaid by slices containedin the one or more payload packets which the respective ROI packetrefers to by way of its immediately preceding the one or more payloadpackets as described above with respect to the “prefixed slices”.

ROI packets may allow for identifying more than one ROI per prefixedslices with identifying the associated tiles for each of these ROIs(c.p. num_rois_minus1). Then, for each ROI, a priority may betransmitted allowing for ranking the ROIs in terms of priority (c.p.roi_priority[i]). In order to allow for a “tracking” of ROIs over timeduring a picture sequence of the video, each ROI may by indexed with anROI index so that ROIs indicated in the ROI packets may be associatedwith each other beyond/across picture boundaries, i.e. over time (c.p.roi_id[i]).

A network entity which receives the video data stream (i.e. a video datastream having video content encoded therein, using predictive andentropy coding, in units of slices into which pictures of the videocontent are spatially subdivided, using a coding order among the slices,with restricting predictions of the predictive coding to the inner oftiles into which the pictures of the video content are divided, whilecontinuing probability adaptation of the entropy coding over the wholeslices, wherein the sequence of the slices in coding order arepacketized into payload packets of a sequence of packets (NAL units) ofthe video data stream in the coding order, the sequence of packets beingdivided into a sequence of access units so that each access unitcollects the payload packets having packetized thereinto slices relatingto a respective picture of the video content) may be configured toidentify, based on the tile identification packets, packets packetizingslices which overlay the tiles which belong to the ROI of the pictures.

The network entity may exploit the information conveyed by the ROIpacket in a manner similar as explained above in this previous sectionregarding the tile identification packets.

With regard to the current section as well as the previous section, itshould be noted that any network entity, such as a MANE or decoder, isable to ascertain which tile or tiles are overlaid by the slice orslices of a payload packet currently inspected, simply by surveying theslice order of the slices of the pictures and surveying the progress ofthe portion of the current picture these slices cover, relative to theposition of the tiles in the picture, which may be explicitly signaledin the data stream as explained above or may be known to encoder anddecoder by convention. Alternatively, each slice (except the first of apicture in scan order) may be provided with an indication/index(slice_address measured in units of coding tree blocks) of the firstcoding block (e.g. CTB) same refers to (same codes) so that the decodermay place each slice (its reconstruction) into the picture from thisfirst coding block on into the direction of the slice order.Accordingly, it may suffice if the aforementioned tile informationpackets merely comprise the index of the first tile(first_tile_id_in_prefixed_slices) overlaid by any slice of theassociated one or more payload packets immediately following therespective tile identification packet since it is clear for the networkentity upon encountering the next tile identification packet in linethat if the index conveyed by the latter tile identification packetdiffers from the previous one by more than one, then the payload packetsbetween those two tile identification packets cover the tiles having thetile index therebetween. This is true if, as mentioned above, both tilesubdivision and coding block subdivision are, for example, based on arow/column-wise subdivision having a raster scan order defined thereamong which is, for both tiles and coding blocks, row-wise, for example,i.e. the tile index increases in this raster scan order as well as theslices follow each other in accordance with the slice order along thisraster scan order among the coding blocks.

The aspect of packetized and interspersed slice header signalingdescribed derivable from above embodiments is also combinable with anyone of the aforementioned aspects or any combination thereof. Thepreviously explicitly described slice prefixes, for example, inaccordance with version 2 unified all these aspects. An advantage of thepresent aspect is the possibility of rendering slice header data moreeasily available for network entities as they are conveyed inself-contained packets external to prefixed slices/payload packets, anda repetitive transmission of the slice header data is enabled.

Accordingly, a further aspect of the present specification is the aspectof packetized and interspersed slice header signaling and may be, inother words, seen as revealing a video data stream having video contentencoded therein in units of sub-portions (see coding tree blocks orslices) of pictures of the video content, each sub-portion beingrespectively encoded into one or more payload packets (see VCL NALunits) of a sequence of packets (NAL units) of the video data stream,the sequence of packets being divided into a sequence of access units sothat each access unit collects the payload packets relating to arespective picture of the video content, wherein the sequence of packetshas interspersed thereinto slice header packets (slice prefix) conveyingslice header data for, and missing in, one or more payload packets whichfollow the respective slice header packet in the sequence of packets.

A network entity which receives the video data stream (i.e. a video datastream having video content encoded therein in units of sub-portions(see coding tree blocks or slices) of pictures of the video content,each sub-portion being respectively encoded into one or more payloadpackets (see VCL NAL units) of a sequence of packets (NAL units) of thevideo data stream, the sequence of packets being divided into a sequenceof access units so that each access unit collects the payload packetsrelating to a respective picture of the video content, wherein thesequence of packets has interspersed thereinto slice header packets) maybe configured to read slice header along with payload data for theslices from the packets with, however, deriving from the slice headerpackets slice header data and skipping reading the slice header for oneor more payload packets which follow the respective slice header packetin the sequence of packets, but adopting the slice header derived fromthe slice header packet which the one or more payload packets follow,instead.

As was true with the above mentioned aspects, it is possible that thepackets, here the slice header packets, may also have the functionalityof indicating to any network entity such as a MANE or decoder, thebeginning of a decoding unit or a beginning of runs of the one or morepayload packets prefixed by the respective packet. Accordingly, thenetwork entity in accordance with the present aspect may identify thepayload packets for which reading the slice header has to be skippedbased on the aforementioned syntax elements in this packet, namelysingle_slice_flag, in combination with, for example,decoding_unit_start_flag, among which the latter flag enables, asdiscussed above, a retransmission of copies of certain slice headerpackets within decoding units. This is useful, for example, as the sliceheader of the slices within one decoding unit may change along thesequence of slices, and accordingly, while slice header packets at thebeginning of decoding units may have the decoding_unit_start_flag set(being equal to one), slice header packets positioned therebetween mayhave this flag not set, so as to prevent any network entity from falselyinterpreting the occurrence of this slice header packet as a beginning anew decoding unit.

Although some aspects have been described in the context of anapparatus, it is clear that these aspects also represent a descriptionof the corresponding method, where a block or device corresponds to amethod step or a feature of a method step. Analogously, aspectsdescribed in the context of a method step also represent a descriptionof a corresponding block or item or feature of a correspondingapparatus. Some or all of the method steps may be executed by (or using)a hardware apparatus, like for example, a microprocessor, a programmablecomputer or an electronic circuit. In some embodiments, some one or moreof the most important method steps may be executed by such an apparatus.

The inventive video data stream can be stored on a digital storagemedium or can be transmitted on a transmission medium such as a wirelesstransmission medium or a wired transmission medium such as the Internet.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, having electronically readable controlsignals stored thereon, which cooperate (or are capable of cooperating)with a programmable computer system such that the respective method isperformed. Therefore, the digital storage medium may be computerreadable.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein. The data carrier, the digital storagemedium or the recorded medium are typically tangible and/ornon-transitionary.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatusor a system configured to transfer (for example, electronically oroptically) a computer program for performing one of the methodsdescribed herein to a receiver. The receiver may, for example, be acomputer, a mobile device, a memory device or the like. The apparatus orsystem may, for example, comprise a file server for transferring thecomputer program to the receiver.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are advantageously performed by any hardware apparatus.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

REFERENCES

-   [1] Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, Ajay    Luthra, “Overview of the H.264/AVC Video Coding Standard”, IEEE    Trans. Circuits Syst. Video Technol., vol. 13, N7, July 2003.-   [2] JCT-VC, “High-Efficiency Video Coding (HEVC) text specification    Working Draft 7”, JCTVC-I1003, May 2012.-   [3] ISO/IEC 13818-1: MPEG-2 Systems specification.-   [4] IETF RFC 3550—Real-time Transport Protocol.-   [5] Y.-K. Wang et al., “RTP Payload Format for H.264 Video”, IETF    RFC 6184, http://tools.ietf.org/html/-   [6] S. Wenger et al., “RTP Payload Format for Scalable Video    Coding”, IETF RFC 6190, http://tools.ietf.org/html/rfc6190-   [7] T. Schierl et al., “RTP Payload Format for High Efficiency Video    Coding”, IETF internet draft,    http://datatracker.ietforg/doc/draft-schierl-payload-rtp-h265/

What is claimed:
 1. An apparatus for decoding a data stream toreconstruct video content, comprising: a buffer; and a decoderconfigured for extracting, from the data stream, a sequence of packetsorganized into a plurality of access units, wherein each of theplurality of access units relates to a picture of the video content andincludes one or more decoding units and each of the one or more decodingunits includes at least one payload packet, and subsets of the sequenceof packets are arranged in slices, the extracting including entropydecoding of the slices across slice boundaries in accordance with awavefront parallel processing technique, identifying, in each of theplurality of access units, a timing control packet corresponding to theone or more decoding units, wherein the timing control packet isindicative of a decoder buffer retrieval time by which content of thecorresponding decoding unit is to be retrieved from the buffer,retrieving, from the buffer, the content including the at least onepayload packet of each decoding unit in accordance with a decoder bufferretrieval time associated with the decoding unit, wherein each packet ofthe sequence of packets includes a packet type field in a packet headerof the respective packet, the packet type field for the at least onepayload packet being different than the packet type field for the timingcontrol packet, and decoding the one or more decoding units usingpredictive decoding to reconstruct the video content.
 2. The apparatusof claim 1, wherein the at least one payload packet is of anon-removable packet type and each of the one or more timing controlpackets is of a removable packet type.
 3. The apparatus of claim 1,wherein the sequence of packets includes Supplemental EnhancementInformation (SEI) packets of different types, each of the SEI packetsincludes an SEI packet type field, wherein content of an SEI packet typefield of a timing control packet differs from that of a different typeof SEI packet.
 4. The apparatus of claim 1, wherein the at least onepayload packet includes encoded information of a sub-portion of apicture of the video content.
 5. The apparatus of claim 1, whereindifferent types of packets in the sequence of packets are arranged inaccordance with a pre-determined organization, and an organization ofthe packets in the sequence of packets can be determined in accordancewith the pre-determined organization when one or more removable packetsare removed from the sequence of packets.
 6. The apparatus of claim 5,wherein the different type of SEI packet includes a Region of Interest(ROI) packet.
 7. An apparatus for encoding video content into a datastream, comprising: a buffer; and an encoder configured for, based onthe video content, encoding video content into a sequence of packetsorganized into a plurality of access units, wherein each of theplurality of access units relates to a picture of the video content andincludes one or more decoding units and each of the one or more decodingunits includes at least one payload packet, and subsets of the sequenceof packets are arrantged in slices, the encoding including entropyencoding of the slices across slice boundaries in accordance with awavefront parallel processing technique, and interspersing, in each ofthe plurality of access units, a timing control packet corresponding tothe one or more decoding units, respectively, wherein the timing controlpacket is indicative of a decoder buffer retrieval time by which contentincluding the at least one payload packet of the corresponding decodingunit is to be retrieved from the buffer, wherein each packet of thesequence of packets includes a packet type field in a packet header ofthe respective packet, the packet type field for the at least onepayload packet being different than the packet type field for the timingcontrol packet.
 8. The apparatus of claim 7, wherein the at least onepayload packet is of a non-removable packet type and each of the one ormore timing control packets is of a removable packet type.
 9. Theapparatus of claim 7, wherein the sequence of packets includesSupplemental Enhancement Information (SEI) packets of different types,each of the SEI packets includes an SEI packet type field, whereincontent of an SEI packet type field of a timing control packet differsfrom that of a different type of SEI packet.
 10. The apparatus of claim7, wherein the at least one payload packet includes encoded informationof a sub-portion of a picture of the video content.
 11. The apparatus ofclaim 7, wherein different types of packets in the sequence of packetsare arranged in accordance with a pre-determined organization, and anorganization of the packets in the sequence of packets can be determinedin accordance with the pre-determined organization when one or moreremovable packets are removed from the sequence of packets.
 12. Theapparatus of claim 11, wherein the different type of SEI packet includesa Region of Interest (ROI) packet.
 13. A method for decoding a datastream to reconstruct video content, comprising: extracting, from thedata stream, a sequence of packets organized into a plurality of accessunits, wherein each of the plurality of access units relates to apicture of the video content and includes one or more decoding units andeach of the one or more decoding units includes at least one payloadpacket, and subsets of the sequence of packets are arranged in slices,the extracting including entropy decoding of the slices across sliceboundaries in accordance with a wavefront parallel processing technique;identifying, in each of the plurality of access units, a timing controlpacket corresponding to the one or more decoding units, wherein thetiming control packet is indicative of a decoder buffer retrieval timeby which content of the corresponding decoding unit is to be retrievedfrom a buffer; retrieving, from the buffer, the content including the atleast one payload packet of each decoding unit in accordance with adecoder buffer retrieval time associated with the decoding unit, whereineach packet of the sequence of packets includes a packet type field in apacket header of the respective packet, the packet type field for the atleast one payload packet being different than the packet type field forthe timing control packet; and decoding the one or more decoding unitsusing predictive decoding to reconstruct the video content.
 14. Themethod of claim 13, wherein the at least one payload packet is of anon-removable packet type and each of the one or more timing controlpackets is of a removable packet type.
 15. The method of claim 13,wherein the sequence of packets includes Supplemental EnhancementInformation (SEI) packets of different types, each of the SEI packetsincludes an SEI packet type field, wherein content of an SEI packet typefield of a timing control packet differs from that of a different typeof SEI packet.
 16. The method of claim 13, wherein the at least onepayload packet includes encoded information of a sub-portion of apicture of the video content.
 17. The method of claim 13, whereindifferent types of packets in the sequence of packets are arranged inaccordance with a pre-determined organization, and an organization ofthe packets in the sequence of packets can be determined in accordancewith the pre-determined organization when one or more removable packetsare removed from the sequence of packets.
 18. The method of claim 17,wherein the different type of SEI packet includes a Region of Interest(ROI) packet.
 19. A machine readable non-transitory medium for storingdata associated with video content, comprising: a data stream stored inthe non-transitory machine readable medium, the data stream comprising asequence of packets, wherein subsets of the sequence of packets arearranged in slices, and the slices are encoded into the data streambased on entropy encoding of the slices across slice boundaries inaccordance with a wavefront parallel processing technique, and thesequence of packets is organized into a plurality of access units,wherein each of the plurality of access units relates to a picture ofthe video content and includes one or more decoding units, each of theone or more decoding units includes at least one payload packet, each ofthe plurality of access units has a timing control packet correspondingto the one or more decoding units, wherein the timing control packet isindicative of a decoder buffer retrieval time by which content includingthe at least one payload packet of the corresponding decoding unit is tobe retrieved from a buffer, and each packet of the sequence of packetsincludes a packet type field in a packet header of the respectivepacket, the packet type field for the at least one payload packet beingdifferent than the packet type field for the timing control packet. 20.The machine readable non-transitory medium of claim 19, wherein the atleast one payload packet is of a non-removable packet type.
 21. Themachine readable non-transitory medium of claim 19, wherein the sequenceof packets includes Supplemental Enhancement Information (SEI) packetsof different types, each of the SEI packets includes an SEI packet typefield, wherein content of an SEI packet type field of a timing controlpacket differs from that of a different type of SEI packet.
 22. Themachine readable non-transitory medium of claim 19, wherein the at leastone payload packet includes encoded information of a sub-portion of apicture of the video content.
 23. The machine readable non-transitorymedium of claim 19, wherein different types of packets in the sequenceof packets are arranged in accordance with a pre-determinedorganization.
 24. The machine readable non-transitory medium of claim23, wherein the different type of SEI packet includes a Region ofInterest (ROI) packet.
 25. The machine readable non-transitory medium ofclaim 19, wherein each of the one or more timing control packets is of aremovable packet type.